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This paper is one of a set of papers, developed simultaneously and presented within a 
single conference session, that are intended to highlight systems analysis and design 
capabilities within the Systems Analysis and Concepts Directorate (SACD) of the National 
Aeronautics and Space Administration (NASA) Langley Research Center (LaRC). This 
paper focuses on the specific capabilities of uncertainty/risk analysis, quantification, 
propagation, decomposition, and management, robust/reliability design methods, and 
extensions of these capabilities into decision analysis methods within SACD. These 
disciplines are discussed together herein under the name of Decision Support Methods and 
Tools. Several examples are discussed which highlight the application of these methods 
within current or recent aerospace research at the NASA LaRC. Where applicable, 
commercially available, or government developed software tools are also discussed. 
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= Systems Engineering 


I. Introduction 

T HE discipline of Systems Analysis (SA) at the National Aeronautics and Space Administration (NASA) 
Langley Research Center (LaRC) encompasses a broad spectrum of analysis and design capabilities devoted to 
aerospace components and vehicles. These capabilities include a wide variety of traditional vehicle analysis 
specialty areas (disciplines) such as aerodynamics, structures, controls, heat transfer, and trajectory analysis. But a 
growing number of people within NASA, and particularly at NASA LaRC, also perform analyses which span across, 
and are distinct from, these specialty areas to support the process of making decisions based upon computational 
simulations. For example, systems integration, multidisciplinary optimization, mission and trade space analyses, life 
cycle cost analyses, uncertainty/risk analysis and management, robust and reliability design methods, technology 
assessments, research portfolio analyses, and “system of systems” architecture analyses all fall into this category of 
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capabilities. This paper is one of a set of papers, developed for presentation within a single conference session, that 
are intended to highlight the SA capabilities of NASA LaRC Systems Analysis and Concepts Directorate (SACD). 
This paper discusses methods and tools in the specific areas of uncertainty/risk analysis, quantification, propagation, 
decomposition, and management, robust/reliability design methods, and extensions of these capabilities into 
decision analysis methods that support the goals and requirements of NPRs 7120. 5 1 , 8000. 4 2 and 8705. 5 3 . For 
convenience, this group of disciplines will simply be referred to collectively herein as Decision Support (DS) 
methods and tools. These DS methods and tools both overlap with, and are distinct from, conventional SA technical 
processes (described subsequently) and fill critical roles in the SA process. 

A companion paper 4 defines SA as the “unique combination of discipline[s] and skills to work in concert with 
NASA Fleadquarters and the NASA Centers to perform studies for decision makers... to enable informed 
programmatic and technical decisions”. The same paper 4 defines risk analysis as “the process of quantifying both 
the likelihood of occurrence and consequences of potential future event.” Figure 1, taken from Ref. 4, illustrates the 
NASA LaRC Systems Engineering (SE) and Analysis process. 



Figure 1. Langley’s Systems Engineering and Analysis Process 


NASA Procedural Requirements, NPR 7123.1, Systems Engineering Procedural Requirements 5 , defines systems 
engineering as “a logical systems approach performed by multidisciplinary teams to engineer and integrate NASA's 
systems to ensure NASA products meet customers needs.” The same document also defines systems approach as 
the application of a systematic, disciplined engineering approach that is quantifiable, recursive, iterative, and 
repeatable for the development, operation, and maintenance of systems integrated into a whole throughout the life 
cycle of a project or program.” This systems approach includes 17 common technical processes, as shown in NPR 
7123. 1 5 , Fig. 3-1 and 3-2, and as follows: 

A. System Design Processes 

1 . Stakeholder Expectations Definition 

2. Technical Requirements Definition 

3. Logical Decomposition 

4. Physical Solution 

B. Product Realization Processes 

5. Product Implementation 

6. Product Integration 

7. Product Verification 

8. Product Validation 

9. Product Transition 
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C. Technical Management Processes 

10. Technical Planning 

1 1 . Requirements Management 

12. Interface Management 

13. T echnical Risk Management 

14. Configuration Management 

15. Technical Data Management 

16. Technical Assessment 

17. Decision Analysis 

The 17 common technical processes are not intended to be performed strictly sequentially. In fact, it is expected 
that, at least, the Technical Management Processes (items C) take place concurrently with the System Design 
Processes (items A) and/or the Product Realization Processes (items B). But any of the processes may take place in 
some combination of sequential and concurrent steps, or all may take place concurrently, as appropriate. 
Furthermore, it is expected that numerous passes through each of the 17 processes may occur, with iteration cycles 
and re-entry points established as appropriate. The reader is referred to NPR 7123.1 5 for detailed discussions of 
each of these common technical processes. 

However, simply utilizing a good SA or SE process, such as that described in NPR 7123. 1 5 does not ensure that 
all the customer’s requirements can be satisfied within cost, schedule, or safety constraints, or with the tools and 
methods available. Likewise, satisfying the customer’s requirements does not mean that the results were obtained 
by a systematic, disciplined engineering approach that is quantifiable, recursive, iterative, and repeatable (as defined 
in NPR 7123. 1 5 ) for the development, operation, maintenance, and disposal of systems. The two aspects of this 
problem, customer satisfaction and good SA/SE process, are really mutually independent, though correlated, as 
shown in Fig. 2, which is fashioned after a typical risk assessment matrix, discussed later in this paper. That is to 
say, an SA or SE process could produce outcomes in any cell of the matrix in Fig. 2, including four corners of the 
matrix, of which the lower left corner is the best outcome possible. However, a good SA or SE process should 
include a negotiation between the developer/provider and the customer, early in project lifetime and throughout, to 
ensure that a reasonable chance exists to satisfy the customer’s requirements within cost, schedule, and safety 
constraints. The customer should be suspect of any results obtained by a poor and/or undocumented process. 
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Figure 2. SA/SE Process Risk Assessment Chart 


NASA Procedural Requirements, NPR 8705.5, Probabilistic Risk Assessment (PRA) Procedures for NASA 
Programs and Projects 3 , states that it “is NASA policy to implement structured risk management (RM) processes 
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and use qualitative and quantitative risk assessment techniques to support optimal decisions regarding safety and the 
likelihood of mission success.” The same NPR states “PRA provides a framework to quantify uncertainties in 
events that are important to system safety. By requiring the quantification of uncertainty, PRA informs the decision- 
makers of the sources of uncertainty and provides information that helps determine the worth of investing resources 
to reduce uncertainty.” Furthermore, the same NPR also states “PRA results are directly applicable to resource 
allocation and other kinds of RM decision-making based on its broader consequence metrics.” It should be clear 
that decisions about resource allocation are implied in Fig. 2, because movement from the upper right-hand corner 
toward the lower left-hand corner of this figure can only be accomplished by the application of additional resources 
above and beyond those of the current situation. Flowever, NPR 8705. 5 3 also states “it addresses technical and 
safety risk and does not address programmatic risk involving consideration of cost and schedule.” Hence, PRA 
methods and tools are often distinct from the DS methods and tools, focusing on Safety and Mission Assurance. An 
accompanying paper 6 discusses analysis techniques other than the PRA. Program/Project Risk Management is 
addressed in NPR 7120.5 and NPR 8000.4. In project management, the additional resources required for 
implementing risk mitigations are ideally identified early and funded out of project resource reserves. 

The DS methods and tools fill three critical roles in the SA process: 1) provide the evaluations of uncertainty, 
risk, and decision-making metrics within the analysis or design process, possibly at all phases of the 17 common 
technical processes above, 2) provide feedback to the system analysis and design processes, and 3) provide feedback 
to decision makers or stakeholders managing the analysis or design process so that they can make informed choices. 
The feedback to the systems analysis or design process may take the forms of adjustments or corrections to the 
constraint and objective metrics within the analysis or design, side constraints on design variables, modifications to 
the analysis/design process, or even path selection within the system analysis or design process. The feedback to 
decision makers or stakeholders may take the forms of uncertainty bounds on deterministic results, probabilities or 
probability distributions, or quantified information about risks. The DS methods and tools can be thought of as an 
overlay to Fig. 1, which attempts to quantify uncertainty and risk at each step in that process. DS is a quantifying 
process relied heavily upon in the Risk Analysis step of Continuous Risk Management. Where applicable, 
commercially available, or government developed software tools are also discussed in this paper. 

Recent experience has emphasized the need for NASA to account properly for the uncertainties that are inherent 
in engineering analyses. For example, during a recent Aeronautics Enterprise Systems Analysis Methods 
Roadmapping Workshop, it was claimed that “by 2025 we need a design and analysis process with known 
uncertainty in the analysis covering the full vehicle life cycle for all aeronautic vehicles (subsonic to hypersonic). 
The process should be variable-fidelity, flexible (customizable, modular), robust (design to confidence level, 
reliable) and validated.” Moreover, the following quotes from the Columbia Accident Investigation Board (CA1B) 
report 7 , clearly illustrate this need: 

The assumptions and uncertainty embedded in this [debris transport] analysis were never fully presented to the Mission 
Evaluation Room or the Mission Management Team. 

Engineering solutions presented to management should have included a quantifiable range of uncertainty and risk 
analysis. Those types of tools were readily available, routinely used, and would have helped management understand the 
risk involved in the decision. Management, in turn, should have demanded such information. The very absence of a clear 
and open discussion of uncertainties and assumptions in the analysis presented should have caused management to probe 
further. 

Likewise, these quotes from the Final Report of the Return to Flight Task Group 8 , Annex A.2, “Observations by 
Dr. Dan L. Crippen, Dr. Charles C. Daniel, Dr. Amy K. Donahue, Col. Susan J. Flelms, Ms. Susan Morrisey 
Livingstone, Dr. Rosemary O’Leary, and Mr. William Wegner” further amplify this point: 

In the case of debris analysis, models for: 1) debris liberation; 2) aerodynamic characteristics of the debris; 3) transport 
analysis of debris; 4) impact tolerance of the thermal protection system; and, 5) the resultant thermal and structural 
models of the effects of damage, are all necessary to assess risk. The uncertainties in one model (or system) inherently 
feeds into and compounds the uncertainty in the second model (or system), and so on. It appears, however, that NASA 
largely designed these five classes of models without the attention to the interdependencies between the models necessary 
for a complete understanding of the end-to-end result. Understanding the characteristics of, and validating and verifying, 
one type of model without examining the implications for the end-to-end result is not sufficient. 

Further compounding the modeling challenge is the fact that the models most often used for debris assessment are 
deterministic, yielding point estimates, without incorporating any measure of uncertainty in the result. Methods exist to 
add probabilistic qualities to the deterministic results, but they require knowledge of the statistical distribution of the 
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many variables affecting the outcome... The probabilistic analysis... is very dependent on the quality of the assumptions 
made by the developers. Although they evaluated some of the assumptions used by the model developers, the... end-to- 
end “peer review” primarily analyzed whether the output of one model could be incorporated into the next, not the joint 
probability associated with any given output. . .without which it is difficult to know the reliability of the result. 

Probability distributions are analytic methods necessary when assessing risk. Without an understanding of the likelihood 
of an outcome, risk acceptance is a judgment based on instinct and experience. But, as the Columbia accident showed, in 
a high risk environment that involves many unknowns like human space flight, experience and instinct are poor 
substitutes for careful analysis of uncertainty. 

II. Uncertainty Analysis / Management 

Uncertainty analysis includes the steps of uncertainty quantification, propagation, decomposition, and 
management, as well as robust design methods and reliability methods. Each of these topics is discussed within the 
following subsections. Numerous aerospace examples of uncertainty analysis / management from SACD and prior, 
similar organizations are available and discussed within the following sections. 

A. Uncertainty Quantification 

Uncertainty quantification 9 ' 14 (UQ) can be used to both improve and clarify the evaluations of the probability of 
an occurrence, and of the consequence of an occurrence within a risk assessment. Uncertainty quantification begins 
in steps 1 and 2 of the 17 common technical processes above, with the examination of requirements for a given 
problem to identify “fuzzy” statements, those that cannot be definitively fulfilled. If the requirements include fuzzy 
statements, the provider should attempt to work with the customer to better define the requirements, or at least 
consider how one will establish whether the requirement is met or not. One should also examine specific numerical 
values given within requirement statements to determine if these numerical boundaries represent laws of physics, 
which cannot be changed, feasible numerical restrictions, or if they are merely desirable limits. If the requirements 
include desirable limits, the provider should attempt to determine how much the customer is willing to pay to 
achieve the desired limit, versus something in the neighborhood of the desired limit. For example, it may be the 
desired limit is only obtainable at very high cost, whereas values in the vicinity of the desired limit may be achieved 
at a substantially lower cost. In all cases, the requirements should be examined to determine if they unambiguous, 
make sense, are achievable, and are not conflicting with other requirements. 

One should also examine the conceptual, mathematical, algorithmic, and computational models (steps 3 through 
5 of the above 17 common technical processes) for physics or behavioral simplifications, temporal and spatial 
approximations, conversions from continuous to discrete analysis, geometric / spatial / temporal resolution, physics 
fidelity, and coding errors which can all introduce uncertainties and/or errors into the process. The executable 
models and data sets that will be used in evaluating the performance, cost, schedule, and safety metrics of interest 
for the problem should be examined for incorrect input data, incorrect usage, and incorrect interpretation of results 
(steps 5 through 9 of the above 17 common technical processes). These examinations should attempt to identify 
inputs to, and parts of, the process that are unknown, variable, or outside of the control of the analyst or designer. 
Having identified these which items contribute uncertainty to the process, one should then attempt to quantify the 
amount of uncertainty associated with each source of uncertainty. This might be as simple as saying there is 
possibly a 10% error in this number, or that a certain input potentially has a lowest possible value, a highest possible 
value, and a most likely value. In general, the more information that can be used to describe the range of the input 
and model uncertainties, and the distributions associated with the input variables or model approximations, the 
better. These input and model uncertainties can then be propagated through the analysis and design tools and 
methods to produce distributions of the output metrics of interest. The uncertainty in the outputs can then be 
decomposed into the most significant and least significant sources, and finally managed. It is important to note, 
however, that the entire uncertainty evaluation, propagation, decomposition, and management process is limited by 
the impact) s) of the most uncertain item(s) within the process. 

Uncertainty quantification is generally a statistical analysis of available input data, related to a particular source 
of uncertainty, for an analysis or design process. The goal of this activity is to determine the type of probability 
distribution to use, and the values of parameters which can be used to define the probability distribution or 
probability density function (PDF), and its accompanying cumulative distribution function (CDF), the integral of the 
PDF. The simplest forms of uncertainty quantification would rely on knowing only a mean value and a standard 
deviation for which one might assume a normal distribution, or knowing only the minimum and maximum values, 
one might assume a uniform distribution over the data range. But the choice of a PDF type should not be arbitrarily 
made, because this choice can have a significant affect on the outcome distribution. Various commercially 
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available tools can be used in this activity, such as Model Center from Phoenix Integration (http://www.phoenix- 
int.com/), the Microsoft Excel add-in called Crystal Ball from Decisioneering, Inc. 
( http://www.decisioneering.com/index.html ), the Microsoft Excel add-in called @Risk from Palisade Corporation 
(http://www.palisade.com/Default.asp) or the PredictionProbe, Inc. tool called Distribution Probe 
( http://www.predictionprobe.com/index.htm ), and many others. These tools generally accept sets of input data, 
allow the user to fit various PDF types to the input data, determine the optimal parameters within a PDF type to fit 
the input data, and rank the resulting distribution fits to allow the user to choose the best PDF to model their data. 
Another commercially available tool called Design Expert from Stat-Ease 15 " 16 , Inc. will perform normality checks on 
the data set to determine if the data can be well modeled with a normal distribution. Ideally, having identified and 
selected a best fit distribution, and perhaps several other possible distribution fits, this set of uncertainty distributions 
should each be propagated through the analysis or design process to determine if the output distribution is sensitive 
to the choice of the input distribution. If great variability is observed in the output distribution as the input 
distribution changes, further validation of the input distribution should be performed, or more input data samples 
obtained, in order to ensure that the input data distribution has been correctly modeled. 

One recent uncertainty quantification effort within SACD involved the use of the Crystal Ball software to 
provide a probabilistic characterization of the uncertainty in launch insertion accuracy and launch date delays. 
Periapse, apoapse, and inclination accuracy data was analyzed from approximately 1 10 combined launches of 
Pegasus XL, Atlas II, and Delta II launch vehicles. Launch delay data for small satellites, other missions, and the 
combined set of small satellites and other missions were also analyzed. Each analysis resulted in a “best fit” PDF, 
by one of several choices of methods. The resulting PDF were then used as inputs to a probabilistic satellite orbital 
lifetime analysis 17 . 

1. Jet Flap Airfoil Uncertainty Analysis 

Another ongoing uncertainty quantification effort within SACD involves the application of a post-processor 
Analysis of Variance (ANOVA) technique ls " 2n within the Design-Expert software (version 6.0.10) from Stat-Ease, 
Inc. 15 " 16 to characterize the aerodynamic performance uncertainty for a jet flap wing configuration proposed for 
advanced military aircraft. The jet flap wing concept would use trailing edge blowing to induce circulation around 
the wing, changing its lift, drag, and pitching moment characteristics 21 . The ANOVA study was conducted to 
quantify the uncertainty of the computational results relative to the experimental results. Analysis of variance is a 
statistical technique that subdivides the total variation of a set of data into component parts associated with specific 
sources of variation. This study proved to be a fairly benign test of the ANOVA capabilities. 

To perform the ANOVA, factors such as the Mach number, blowing coefficient, angle of attack, and a grid 
density metric, which potentially affect the computed lift and pitching moment coefficient, compared to the 
measured data, were identified and input to the software. Ranges of interest for the factors were input to the 
software along with the discrete computed or measured values of the factors. Also input were the computed or 
measured lift and pitching moment coefficient associated with the input values of the factors. In a matter of 
seconds, the Design-Expert software calculated a numerical model of user-specified order (linear, 2-factor 
interference, quadratic, or cubic) to fit the input data, residuals (differences) between the model and actual data were 
produced at each of the input conditions, and uncertainties (in the form of Least Significant Differences or LSD) for 
experimental, computational, and combined experimental / computational data sets were computed. The LSD 
computed by the software indicate the smallest resolvable differences in the functional values (lift and pitching 
moment coefficient) given just the input data points from selected data sets. The software also provides a collection 
of diagnostics which evaluate the suitability of the input data set for use within the ANOVA process and which 
examine the behavior of the resultant data, suggesting transformations which should be applied to the data to reduce 
the LSD. 

Figures 3-6 illustrate some of the key features and results from the uncertainty analysis studies. Figure 3 
illustrates the uncertainty analysis for the computational grid sensitivity study, described in Ref. 21 at Mach=0.3. 
The curve in the plot represents the approximate quadratic numerical model, which the Design-Expert software 
fitted to the input data. Design points are shown where input data were provided. As in Ref. 21, Fig. 6, the lift 
coefficient is shown as a function of the grid density metric, but here with 95% confidence least significant 
difference (LSD) error bars added. The blue square simply indicates where on the curve the LSD is applicable. The 
LSD bars indicate the smallest resolvable differences in the functional values (lift coefficient) attributable to changes 
in the grid density metric, given just the selected input data points. In this case, the LSD is about 0.0050, which is 
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Figure 3. Uncertainty Analysis, 

Grid Sensitivity Study at Mach=0.3, Alpha=6.0°. 


Figure 4. Uncertainty Analysis, 

Grid Sensitivity Study at Mach=0.3, Alpha=6.0°, 
finest grid data excluded. 


divided by two and added or subtracted from the calculated numerical model value to obtain high and low 95% 
confidence bounds. This means the potential lift error due to grid density is about +/-0.0025. On the finest grid, the 
actual lift coefficient at this grid density (0.7750) is expected to be between 0.7724 and 0.7775 with 95% 
confidence; there is a 5% chance that the actual lift on the selected grid could be outside of these bounds. The actual 
lift coefficient, in this case, is well within the 95% confidence bounds. The actual lift coefficient value on the 
selected grid (0.7741, second from left) also lies well within the 95% confidence interval for that grid, where the 
approximate model value is 0.7744, and the range is between 0.7719 and 0.7769 with 95% confidence. However, in 
this example, because of the clearly parabolic shape of the approximate model curve shown in Fig. 3, and the fact 
that several input values of lift coefficient are between 0.7700 and 0.7800, the confidence interval is broader than 
might be expected and is also broader than desired. As shown in Fig. 4, the uncertainty can be reduced on the 
selected grid, indicated by the blue square, simply by ignoring the finest grid results; this removes ambiguity about 
the behavior of the lift coefficient as the grid density is increased (moving toward left on the plot), and allows for a 
prediction of the actual lift coefficient on the selected grid to be within +/- 0.0004 of the approximate model value 
0.7741, or that the actual lift coefficient is expected to be between 0.7737 and 0.7745, with 95% confidence. This is 
a reduction of the prediction uncertainty by more than a factor of ten, by simply removing some ambiguity in the 
data presented to the software. Similar results for the grid sensitivity study at Mach=0.8 (not shown), again 
excluding the data from the finest grid, resulted in a 95% confidence interval (LSD) of about 0.0070. 

Figure 5 illustrates a different way to perform the grid sensitivity uncertainty analysis. In this example, both 
Mach=0.3 and Mach=0.8 have been considered together, with data from the finest grid again excluded. The lift 
coefficient was described to the Design-Expert software as a function of the numerical (continuous) grid density 
factor and a categorical factor of Mach number, which could only take on the discrete values of 0.3 or 0.8. In this 
case, since the lift behavior with grid density for both grid studies has the same functional form, the combination of 
the data allows the software to better resolve the potential error due to grid density. The maximum LSD of the 
combined data set is now about 0.0018, meaning the potential lift coefficient error is only about +/-0.0009 due to 
grid density. 

Figure 6 illustrates the uncertainty analyses for two similar, but different, input data sets. Both computational 
and experimental Mach=0.3 data with blowing coefficient up to about 0.07 are analyzed as a function of the blowing 
coefficient. The difference between computational and experimental data is treated again as a categorical factor, 
which can only take on two distinct values (i.e., Type = Computational or Experimental), whereas the blowing 
coefficient is treated as a numerical (continuous) factor. In this example, repeated experimental data was included, 
only incremental jet effects (Jet On - Jet Off) are considered, and a power transformation applied to all the input lift 
coefficient data (0.00684186 was added to each input lift coefficient value and the sum, for each input data point, is 
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then raised to the 1.63 power), as suggested by the software. This analysis parallels the lift augmentation behavior 
as a function of the blowing coefficient, reported in Ref. 21, Fig. 18, and now analyzed in this transformed space. 
The LSD is about 0.0113 in the transformed space, which can be shown to reduce the LSD to about 0.0067 in the 
untransformed space. The same data (repeated experimental points, and incremental jet effects) analyzed in the 
untransformed space (not shown) yielded an LSD of about 0.0838 - a significantly larger uncertainty band. 
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Figure 5. Uncertainty Analysis, 

Grid Sensitivity Study at Mach=0.3 and Mach=0.8, 
finest grid data excluded. 


2. Hurricane Track Prediction Uncertainty’ Analysis 

The ANOVA technique 15 " 16, 18-20 was also applied 
to two much more challenging data sets. In the first 
example, ANOVA was applied to historical hurricane 
track data. This was presumed to be a very 
challenging test of the ANOVA method and was 
conducted to determine if the technique could 
compete with, or improve upon, current hurricane 
track forecasting methods. Hurricane track and 
intensity data, dating from 1851 through the 
beginning of 2006, was obtained from the U. S. 
Department of Commerce, National Oceanic and 
Atmospheric Administration (NOAA), National 
Hurricane Center (NHC), website 

(http://hurricane.csc.noaa.gov/hurricanes/download.ht 
ml) and loaded into Microsoft Excel for pre- 
processing. The pre-processing step consisted of 
simply rearranging the data in proper format for use 
with the Design-Expert software, and a few minor 
calculations, such as creating a time stamp for from 
the year, month, and date entry of each record. A 
total of 32756 recorded measurements, a subset of the 
data back to only 1885 (the complete data set was too 
large for current version of the software), taken at six-1 


Figure 6. Uncertainty Analysis, Blowing Coefficient 
Sensitivity Study at Mach=0.3, Alpha=6.0°, repeated 
experimental data included, power transformation 
applied, k= 0.00684186, ?i=L63. 



Figure 7. North Atlantic Basin Hurricanes, 

Sized by the Saffir-Simpson Hurricane Scale. 
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Basin hurricanes, tropical storms, and tropical depressions, were provided to the Design-Expert software for analysis 
in this demonstration. Sample hurricane track data only (as discrete points) are shown in Fig. 7, along with a crude 
approximation to the eastern seaboard of the United States of America. The point coloring in Fig. 7 indicates the 
month of the year, and the point size is correlated with the Saffir-Simpson scale ( http://en.wikipedia.org/wiki/Saffir- 
Simpson Flurricane Scale ). 

The storms analyzed were not chosen in any particular way and thus included a wide variety of storm tracks. For 
example, data analyzed include the path of Tropical Storm Josephine (Oct. 1996) and that of Flurricane Lili (Dec. 
1984), as shown in Figure 7. The path of Tropical Storm Josephine is highlighted in purple with large connected 
circular symbols; the storm formed near the western edge of the Gulf of Mexico and moved steadily to the North 
East, and happened to pass very close to Hampton, VA. This storm had the western-most starting point of all the 
storms analyzed. The path of Hurricane Lili is highlighted in Fig. 7 in pink with large connected square symbols; 
the storm formed in the mid North Atlantic, moved to the South and East, circled back to the North, then finally 
moved to the South and West. 

Several factors were chosen from data available that could potentially influence the track and intensity of 
hurricanes as they develop and progress. The factors chosen for this demonstration were: 1) a year fraction 
computed from the month and day, divided by the maximum number of days in the year of record - provides a 
timeline through a typical year to track cyclic behavior on a seasonal basis, 2) the duration of the storm as measured 
in six-hour increments, 3) the starting date including the year, month, and day of the month - provides a timeline 
through all the records to track cyclic behavior on an annual basis, 4) the starting longitude value, 5) the starting 
latitude value, 6) the starting maximum wind speed, 7) the current longitude value, and 8) the current latitude value. 
The responses for this demonstration were the actual hurricane longitude and latitude values 24 and 48 hours after 
the input record. No information was available within this data set regarding local atmospheric or water 
temperatures, or short duration weather patterns which would logically be expected to influence the storm 
development and path. The goal of this exercise was to determine if the software could be used to discern 
correlations and interactions among the factors and seemingly chaotic response data to make an accurate prediction 
of the storm location 24 to 48 hours in advance. The stated goal of the NOAA/NHC is an average (50% confidence 
interval) 48-hour track forecast prediction error of no more than 125 nautical miles (nm), to be achieved by 2009. 
Their achieved average 48-hour track forecast prediction error was just 107 nm in 2003. 


DESIGN-EXPERT Rot 


Interaction Graph 


Figure 8 illustrates the interactions of the current 
longitude and latitude (as two of the eight interacting 
factors within the model) on the 48-hour prediction of 
longitude and latitude, within a transformed space 
suggested by the software (a square root transformation 
with offset of 120.23). The curves are the approximate 
model generated by the software by fitting a modified 
quadratic to the input data at each of the extreme input 
values of current latitude, for given values of current 
longitude (across x-axis), with the other six factors fixed 
at the values shown in the plot. That is to say, for a 
given current longitude between 12° and -109.3°, one can 
interpolate the least significant difference (LSD) for the 
48-hour predicted longitude between the two curves for a 
given current latitude between 7.2° and 69°, and with the 
other six factors fixed at the values shown. The circle 
represents an actual input data point, in this case, the last 
point considered in the path of Tropical Storm Josephine 
(current longitude = -15.5° and current latitude = 55.5°). 
The error bars are the LSD measure, defined previously. 
In this case, the maximum LSD for the conditions of 
interest is 0.220864 in the transformed space, which can be shown to translate into a 95% confidence interval about 
392.1 miles (340.7 nm). That is to say, given the data analyzed, there is only a 5% chance that the actual 48-hour 
longitude predicted location would be outside of these bounds. A similar point (95% confidence) from the 
NOAA/NHC web site (http://www.nhc.noaa.gov/verification/verify2.shtml) yields about a 300 nm 48-hour forecast 
error. The Design-Expert / ANOVA track error 50% confidence interval would be about 1 17.2 nm, compared to the 
NOAA/NHC stated goal of 125 nm from above. Although the Design-Expert / ANOVA error estimate at 95% is 
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about 13% greater than a comparable number from NOAA/NHC, the 50% confidence intervals are comparable. The 
Design-Expert / ANOVA estimate was obtained very quickly, with a desktop workstation, and with no attempt to 
characterize the storm tracks. It is thought that this demonstration represents an encouraging start for further 
investigation. The 48-hour latitude prediction exhibits definite non-normal behavior that would require further 
investigation. 

3. Solar Flux Intensity’ Prediction Uncertainty’ Analysis 

Another application of the ANOVA technique 15 " 16 ’ 18 " 20 , still in progress, is to determine if the Design-Expert 
tool can be used to aid in the long-range (10-50 years) prediction of solar flux intensity 17 . Numerous efforts were 
under taken using a variety of techniques, as the problem proved to be quite challenging. Measured data was 
provided from each month for the period starting with January 1953 and ending with August 2005. Measured solar 
flux intensity data was assumed to be exact, having zero measurement error, since no source to quantify the 
measurement error was readily identified; this assumption is merely a convenience, not a requirement for the 
following analysis. In this case, a simple, 3-point average smoothing scheme, given in Eqs. (1), 

smooth(n) = (measured(n-l) + measured(n) + measured(n+l))/3 

smooth( 1) = measured) 1) (1) 

smooth(np) = measured(np) 

was applied to the measured flux data for all interior points, n = 1, np-1, where np is the number of measured data 
points (633). The smoothing scheme was applied for 91 iterations, until the gross character of the measured data 
was captured by the smoothed data, without all the high frequency variation present in the measured data, as shown 
in Fig. 9. 


Flux + Approximation 



Year 

Figure 9. Measured and Smoothed Solar Flux Intensity 

The variations of the measured data (Flux) about the smoothed data (Smooth3) in Fig. 9 should naturally take on 
a normally-distributed character, with a mean value of zero, by virtue of the smoothing process applied. The 
calculated standard deviation was about 11.42. The normality assumption can be verified by analyzing the deltas 
(Delta = Flux - Smooth3) as a function of real number Year, which is the sum of an integer measurement year, and a 
fraction of a measurement year (for months 1-12, divided by 12) within the Design-Expert software, as shown in 
Fig. 10. In this case, the software suggested that a square root transformation, with a constant of 29.1863 added to 
each input Delta to ensure a positive-definite response, be applied to the input delta data, which has been done. The 
approximately linear behavior of the input data over most of its range in Fig. 10 indicates that the data can be 
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reasonably assumed to be a normally-distributed about the mean value. 


DESIGN- EXPERT Rot 
Sqrt(Delta + 29.19) 


03 

_Q 

2 

Q_ 



Normal Plot of Residuals 



Studentized Residuals 


Figure 10. Normality Assessment of Measured - 
Smoothed Solar Flux Intensity Deltas 

The smoothed data is still a highly complex function of varying amplitude, with multiple frequency and phasing 
components present. An attempt to optimally model the smoothed data function with up to ten sine wave 
components, each having a variable amplitude, frequency, and phase angles (i.e., 30 design variables), proved to be 
unacceptable. However, by rearranging the smoothed data to be Year as a scatter plot function of Smooth3, as 
shown in Fig. 11, the periodicity of the certain points within the smoothed data set could then be easily analyzed. A 
truly periodic function would have a linear increase in the year associated with local minima or maxima, as a 
function of the cycle being investigated. Having rearranged the data in this fashion, it was a simple matter to 
determine the Y ear for the first M points near each local minima or maxima. 


Smooth3 



Flux 

Figure 11. Years Corresponding to Smoothed Solar Flux Intensities 
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Thus, the Year associated with each of the minimum points for the five local minima in Fig. 11 could be input 
into the Design-Expert software as a function of the cycle number to determine if the points exhibited a linear 
behavior. Similarly, the two, three, or eight. Years associated with most-minimum points of the five cycles (or the 
one, two, three, or eight Years associated with the most-maximum points of the five cycles) could also be analyzed 
to determine if the linear relationship improved or degraded, with repetitive entries for the extreme points of each 
cycle. This allows for the possibility that the some measurement error, either in time or intensity, of the extreme 
points might occur. In fact, the local minima and maxima exhibited highly periodic behavior, which improved with 
repetitive entries for each extreme point, as shown in Fig. 12. However, the frequency and phasing of the minimum 
points was slightly different than that of the maximum points. This leads to the shape changes which depart from 
true sine wave for the smoothed data. Specifically, the cycle minima were well approximated by the Design-Expert 
software, with eight points provided at each minimum, by the following periodic model 

Smooth3 -Min = +1943.56671+10.62499* Cycle (2) 

Similarly, the cycle maxima were well approximated by the Design-Expert software, with eight points provided at 
each maximum, by the following model 

Smooth3-Max = +1947.87502+10.81665* Cycle (3) 

The reader should note that the LSD in the figure is about 0.3457, which implies that minimum and maximum points 
are known to within about +/- two months within a 10.6 to 10.8 year cycle. Employing the periodicity implied by 
the above two relationships, numerous estimates of the Year associated with each minimum and maximum can be 
computed. The frequency and phase shift effects implied by the above realtionships determine sets of earliest and 
latest possible start dates of the five cycles of measured data, as shown in Table 1. 


design-expert Rot One Factor Plot 



Table 1. Earliest and Latest Solar Flux Intensity 
Minima and Maxima 



Earliest 

Latest 

Mini 

1953.254 

1954.192 

Maxi 

1958.667 

1959.504 

Min2 

1964.079 

1964.817 

Max2 

1969.492 

1970.133 

Min3 

1974.904 

1975.450 

Max3 

1980.317 

1980.767 

Min4 

1985.729 

1986.083 

Max4 

1991.142 

1991.400 

Min5 

1996.550 

1996.717 

Max5 

2001.958 

2002.033 


A: Cycle 


Figure 12. Actual and Approximate 
Periodicity of Smoothed Solar 
Flux Intensity Minima 

Having determined the earliest and latest Years for which the smoothed minima and maxima data exhibited 
periodic behavior, the measured flux data for each of the five cycles could then be extracted from the original data 
set at these points, and overlaid. Then, average cycle data could be computed from the existing five cycles of data, 
along with deltas of the measured data from the average cycle data. Local standard deviations of the deltas for 
measured data relative to the average cycle data, as a function of Time through the cycle (ranging from a period of 
10.625 to 10.817 years, depending on the Design-Expert model chosen), could then be statistically analyzed with the 
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Design-Expert software. The deltas of measured data relative to the average cycle data were found to be normally 
distributed about the mean values (the average cycle data) to a high degree of confidence. The variations between 
the measured flux data and average cycle data were almost entirely captured within +2 sigma (local standard 
deviations) and -1 sigma of the average cycle curve data, as shown in Fig. 13, which is just one sample (but quite 
representative) of the many possible cycle extractions and overlays. 


Overlayed Cycles with Average and +2/-1 Local Sigma Bounds 



Cycle 1 

Cycle2 

Cycle3 

Cycle4 

Cycle5 

Average 

+2 StDevL 

-1 StDevL 


Time 


Figure 13. Overlapping Measured and Average Solar 
Flux Intensity with +2/-1 Local Standard Deviation Bounds 


Thus, to a high degree of confidence (about 95%), predictions about future solar flux intensity measurements can 
be made by simply adding a normal distribution of delta fluxes with a mean value of zero, and a prescribed standard 
deviation (StDevL) as a function of time throughout a cycle, to Average cycle flux data determined from 50+ years 
of measured data. Although the variation from a possible minimum predicted flux to a possible maximum predicted 
flux at some future time is quite large, the confidence level associated with this prediction is quite high. A 
conservative estimate would be to simply to use the average cycle data plus two times StDevL(t) for a given time in 
a future cycle as a best guess about the future flux intensity. It is expected that this prediction technique is valid for 
at least 50 years into the future, at which point the frequency and phase shift effects present in determining the 
average cycle data and the earliest and latest start dates of cycles may become significant enough to reduce the 
confidence associated with the prediction. 


B. Uncertainty Propagation 

There are numerous methods to perform uncertainty propagation through an analysis or design process. These 
can be generally classified as First- or Second-Order Reliability Methods (FORM/SORM), simulation methods, 
importance sampling methods, response surface methods, and mean value based methods, using the language of the 
PredictionProbe, Inc. tool UNIPASS (http://www.predictionprobe.com/index.htm). The FORM and SORM 
methods are good when the there is only a very small expected probability of failure, and differ only in whether a 
linear or quadratic approximation is used to approximate the constraint boundaries at the most probable point(s) 
(MPP) of failure. Simulation, sampling, and mean value based methods work best when there is a large expected 
probability of “failure” and with many analysis or design input variables; in this case, the failure mode is generally 
thought of as degraded performance, rather than a catastrophic failure. Flence, these methods are suitable for robust 
design applications where a typical goal is to minimize the output sensitivity to variation of the uncertain inputs. 
Response surface methods are generally used when the objective or constraint functions are very expensive to 
evaluate, and hence only a limited few “high fidelity” answers representing the real world behavior are available. 
Several of these classes of examples will be discussed in following sections of this paper. 
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1. Uncertainty Propagation in Stability and Control (S&C) Derivative Application 

One study of uncertainty propagation involved the application of the automatic differentiation (AD) of 
FORTRAN (ADIFOR) tool, version 2.0D 22 ' 27 to the Panel Method Ames Research Center (PMARC) code 28 version 
14.10, a low-order panel method CFD code that was modified by applying AD to enable efficient computation of 
exact first and second force and moment derivatives with respect to a wide variety of code inputs. ADIFOR 2.0D 
was applied to PMARC by using a wide variety of input variables as potential independent variables of 
differentiation. The PMARC body and wind axis forces and moments were used as the dependent variables of 
differentiation. Since uncertainty estimates of all S&C parameters (forces and moments and their derivatives) are 
normally requested by most control law designers, the computation of both first and second derivatives with respect 
to selected independent variables of differentiation are demonstrated; the latter are used to enable the propagation of 
known or assumed input variable uncertainties through the code to determine the uncertainty effects on the 
computed forces and moments. The relationship for uncertainty propagation 11 ' 12 as a function of normally 
distributed, random variables is given in Eqs. (2), as follows: 


X 

i=l 


' c)F / 


,j = l,n 


in = number of random inputs 


n = number of output functions (4) 


<J i = standard deviation of 
the random inputs 


In this case, the output function F may be any of the computed body or wind axis forces and moments, or their 
first derivatives with respect to parameters such as the angle of attack, angle of sideslip, and the steady rotational 
rates. As described in Ref. 13, the original and ADIFOR-generated PMARC code versions were executed for an 
array of oscillatory and rotary aircraft motions to compare with data generated for a 10% scale F-16XL model 
tested 29 in the NASA LaRC 14- by 22-Foot Subsonic Tunnel. 

Derivatives of the forces and moments and their first derivatives with respect to angle of attack were 
differentiated again with respect to angle of attack (a) and a constant pitch rate, Q (or q). Uncertainties with respect 
to the angle of attack were computed and propagated through the code, proportional to variable gradients with 
respect to the uncertain variable, via Eqs. 4. The results from this study are shown in Table 2. The table shows the 
variable name, the uncertain variable name, the computed output variable sigma (from Eq. 1), and one-, two-, and 
three-sigma (representing 84.1, 97.7, and 99.9% , respectively) low/high uncertainty bounds for each variable, as 
well as the mean value for each variable. Table 2 illustrates a typical preliminary uncertainty analysis for the F- 

16XL configuration. The uncertainties in C L , — 1 and — - (with a in degrees and q in degrees per second) 

da dq 

are presented as calculated from first and second derivatives of C L with respect to a and q , assuming cr. =0.1 
for both a and q . The same capability can be easily applied to all of the forces and moments in both the wind and 
body axis systems, for any of the independent variables of interest. 


Table 2. Sample uncertainty analysis for F-16XL all values of a., =0.1 . 


Variable Name 

Uncertainty 

Variable 

Computed 

sigma 

99.9% 

Low 

97.7% 

Low 

84.1% 

Low 

Mean 

Value 

84.1% 

High 

97.7% 

High 

99.9% 

High 

dCL/dALPDEG 

ALPDEG 

4.72E-05 

2.02E-02 

2.03E-02 

2.03E-02 

2.04E-02 

2.04E-02 

2.05E-02 

2.05E-02 

dCL/dALPDEG 

Q 

4.97E-05 

2.02E-02 

2.03E-02 

2.03E-02 

2.04E-02 

2.04E-02 

2.05E-02 

2.05E-02 

dCL/dALPDEG 

ALPDEG+Q 

6.85E-05 

2.02E-02 

2.02E-02 

2.03E-02 

2.04E-02 

2.04E-02 

2.05E-02 

2.06E-02 

CL 

ALPDEG 

2.04E-03 

3.95E-01 

3.97E-01 

3.99E-01 

4.01 E-01 

4.03E-01 

4.05E-01 

4.07E-01 

dCL/dQ 

ALPDEG 

4.97E-05 

2.39E-02 

2.40E-02 

2.40E-02 

2.41 E-02 

2.41 E-02 

2.42E-02 

2.42E-02 

dCL/dQ 

Q 

1.23E-03 

2.04E-02 

2.16E-02 

2.28E-02 

2.41 E-02 

2.53E-02 

2.65E-02 

2.78E-02 

dCL/dQ 

ALPDEG+Q 

1.23E-03 

2.04E-02 

2.16E-02 

2.28E-02 

2.41 E-02 

2.53E-02 

2.65E-02 

2.78E-02 

CL 

ALPDEG 

2.41 E-03 

3.94E-01 

3.96E-01 

3.98E-01 

4.01 E-01 

4.03E-01 

4.06E-01 

4.08E-01 

CL 

TOTAL 

3.15E-03 

3.91 E-01 

3.95E-01 

3.98E-01 

4.01 E-01 

4.04E-01 

4.07E-01 

4.10E-01 
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2. Uncertainty Propagation in the NASA LaRC Conceptual Design Shop (CDS) 

In the past five or six years, the aeronautics group within SACD has partnered with the Aerospace Systems 
Design Laboratory (ASDL) at the Georgia Institute of Technology for some of its risk analysis needs 30 " 33 . The most 
relevant examples are the technology assessments (described subsequently), technology forecasting and technology 
metrics tracking ASDL performed on a yearly basis for the former Ultra Efficient Engine Technology (UEET) 
project, and for the more recent Vehicle Systems Program (VSP). As a parallel effort to that at Georgia Tech, a large 
multidisciplinary team was formed at NASA LaRC to develop the Conceptual Design Shop (CDS) as a framework 
in which one could “efficiently and confidently design and assess atmospheric vehicle concepts and advanced 
technologies to meet NASA's aeronautics goals.” [CDS Scope Document]. It was of special interest to handle 
unconventional configurations and advanced technologies for which previous data and experience is lacking. 
Because uncertainty estimates of S&C parameters (forces and moments and their derivatives) are normally 
requested by most control law designers, the stability and control sub-team within CDS first recognized and 
advocated for developing an in-house uncertainty management and risk analysis capability within CDS. This 
advocacy led to the development of a new Matlab-based tool called the Confidence Module (CM) 34 . 

The CM was to provide the framework and tools for uncertainty management, i.e. quantification, propagation, 
decomposition and accommodation, in the analysis and design of aircraft at the conceptual level. It was of special 
interest to support multidisciplinary variable-fidelity analysis and design environments, and to enable trade-off 
studies evaluating system performance, robustness and reliability. The five year plan was to create a tool which was 
capable of performing uncertainty quantification, propagation, inverse design, decomposition and robust design. 
Unfortunately, CDS was only funded for two years, and only the uncertainty quantification and propagation 
capabilities were actually implemented. 




Propagation Method 

HYBRID: 


Problem: Pf 
Studying Response; 1 
Failure Boundary: 0.7 



R»*pom* Function 


Output 

Response Surface: 



Sampling: hss _ 

RUN' 

- Standard Output 

Fitting Tech: 



polyfit 




Options Window Help 

* x A A 


Random, Normal, 


Number Name Type Distribution Parameters 

a Random Batson Beta 0.4 2.5 10 

b Random Beta 2 2 


Figure 14. CM Graphical User Interface 



Figure 16. PDF and CDF GUI 


Figure 15. Uncertainty Quantification for Input 
Parameters 

Figure 14 shows the main CM graphical user interface, 
written in Matlab®. The display is like a flowchart 
displaying the steps the user has to take to solve a 
probabilistic analysis problem. The bottom left box, is the 
uncertainty quantification box, where the input 
parameters and their associated probability distributions 
are defined. The top left box is where the user chooses the 
uncertainty propagation method he/she wishes to use. 
Finally, the two boxes link to a third box, which is where 
the user chooses whether he/she wants to run the analysis 
code itself, or run a response surface representation of it. 
Once the three are defined the “run” button can be clicked 
and the output is in the form of a “probability of failure” 
metric. The user defines ahead of time what is the target 
or maximum threshold for the response of interest, and 
the CM calculates, what the probability is of exceeding 
that threshold. 
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Figures 15 and 16 above show the uncertainty quantification GUIs. Figure 15 allows the user to add an infinite 
amount of uncertain input parameters and Fig. 16 allows the user to pick the appropriate PDF or CDF describing 
each uncertain parameter. The distribution choices contain all Matlab 8 statistical toolbox distributions (beta, normal, 
uniform, weibull, chi-square, etc), and three customized ones which include the triangular distribution, a generalized 
beta, and the Robert Batson ' distribution, mentioned above. 

The uncertainty propagation methods were coded in Matlab 8 , and include sampling techniques (Monte Carlo 
simulations, Hammersley Sequence Sampling and Latin Hypercube), a first order reliability method (FORM), and a 
hybrid method, which uses the HSS sampling combined with the FORM technique, and uses the appropriate one 
depending on the magnitude of the probability of failure. 

The response surface method currently included in the CM is a polynomial fit, and provides actual versus 
predicted as well as response versus individual input parameter plots, in order for the user to verify whether the fit is 
good. The CM has not been used for any real world problems yet, but has been applied to several sample problems, 
and has proven to be validated. 


C. Uncertainty Decomposition 

Uncertainty decomposition is the process of allocating rolled-up output uncertainties back to significant, primary 
sources. This process usually involves some kind of sensitivity analysis, which may be performed manually by 
trade studies, or by automated tools, such as the ADIFOR 3.0 tool' 6 " 41 which takes standard Fortran code and a 
specification of independent and dependent variables to produce direct (forward-mode) or adjoint (reverse-mode) 
Fortran code that is compiled and executed to evaluate the gradient and/or Hessian matrices of the simulation code, 
in lock-step with the function evaluation. 

NASA’s Vehicle Systems Program in 2004-2005, and other NASA projects such as the Ultra Efficient Engine 
Technology 33 "’ 4 in the past, have asked Georgia Tech’s ASDL to help with quantifying the uncertainty and its 
impact, in their technology assessments. Technology assessments are performed to determine if the technologies 
invested in, will support and meet the project goals, and to identify which technologies have the greatest impact on 
those goals. The assessment provides useful information to enable technology portfolio management and budget 
allocation decision making. 

The assessment is traditionally performed by modeling a baseline vehicle and changing a few parameters to 
reflect the impacts of a particular technology. The process is repeated for multiple technologies individually, and 
then combined, provided that they are compatible. The impacts of each technology are gathered through interviews 
with the researchers and technologists who are familiar with the technology of interest. The results of the interview, 
or technology audits, are then translated into appropriate parameters within the analysis model. ASDL calls these 
parameters, technology k-factors or technology metrics. Each technology can therefore be modeled in the analysis 
code, as a vector of k-factors. However, the values for these metrics are uncertain. They are based on estimating 
methods of varying levels of fidelity, sometime even based on the technologist’s “best guess”. Hence, in order to 
capture this uncertainty, the technology audit process asks the technologist to provide four values for each 
technology metric: minimum, most likely and maximum value, as well as the confidence in the estimate, on a scale 
from one to five. The four values are then translated into a probability density function (PDF), using a technique 
proposed by Dr. Robert Batson 36 , as shown in Fig. 17. 


The Georgia Tech ASDL uncertainty analysis is 
based on three steps, uncertainty quantification, 
propagation, and decomposition. The translation of 
the technologists inputs into a beta distribution 
constitutes the uncertainty quantification step. Now 
that the uncertainty in the inputs is quantified, the 
next step is to estimate how this uncertainty 
propagates through the analysis model, and how it 
affects the uncertainty in the results. This is performed by running a sampling technique called Monte Carlo 
Simulations, using the commercial tool Crystal Ball 8 . However, Monte Carlo simulations require numerous runs 
and can take the analysis code days to run. Hence, instead of propagating the uncertainty through the analysis code 
itself, the Monte Carlo simulations are performed on response surface models of the analysis code. Each response of 
interest (Takeoff gross weight, fuel consumption, etc.) is expressed as a function of k-factors, in the form of a 
quadratic model as shown in Eq. (5). 



Worse Metric Value Better 

Figure 17. Probability Density Function 
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R = b o + X b i k i + Z b n k < + Z Z b v k ‘ k J (5) 

1=1 1=1 i=l 7=/+l 

Figure 18 shows a schematic of the process. Each technology, 1, 2 and 3 have a vector of k- factors, representing 
its impacts, with quantified uncertainty, on the whole vehicle. These technologies are compatible and evaluated 
together. The impacts and uncertainties are modeled and propagated through the vehicle’s mission, and performance 
results are computed with its associated uncertainty. In Fig. 18, the response is shown in the form of a cumulative 
distribution function, which lets the user read off what confidence there is associated with a particular value. 
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Figure 18. Probabilistic Assessment of a Technology Combination 


Uncertainty Decomposition 



Technology K-factors 


Figure 19. Pareto Analysis 

Approach Speed 
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The advantage of using Response Surface 
Methods is that you obtain many different 
sensitivity and statistical data in the process of 
finding a good fit to the analysis model. For 
instance, the commercial statistical package 
J M P " , used to generate the response surfaces 
allows the user to view a Pareto plot, which 
shows which inputs contribute to the 
variability in the response and by how much. 
This constitutes uncertainty decomposition. 
Therefore if the user’s response is not 
satisfactory, the confidence is too low or the 
variability is too high, then the user can 
determine which inputs to focus on, in 
attempting to reduce the uncertainty in the 
problem. The generic Pareto plot in Fig. 19 
illustrates that the response’s variability is 
mainly due to the uncertainty in the TIT, Fan 
efficiency and HPC efficiency parameters. 
More knowledge on those three input 
estimates would help reduce the uncertainty in 
the response. 

Once all the technologies are modeled 
individually and combined, the user obtains a 
comparison between the various technologies 
and how they contribute to the vehicle’s 
performance, and project goals, as shown in 
Figure 20. 


Figure 20. Sensitivity Studies and Technology Assessments 
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D. Uncertainty Management 

Uncertainty management is the multidisciplinary process of "living with” or accommodating identified 
uncertainties. In many ways this is somewhat like the use of safety factors, which have been historically 
incorporated into good engineering analysis and design practices since humans began to build thing that could fail. 
The difference is that with safety factors, it is difficult or impossible to quantify how changing an analysis or design 
variable affects the margin of safety, whereas the goal of uncertainty analysis, propagation, decomposition, and 
management is specifically to quantify these effects. Uncertainty analysis may be incorporated into analysis and 
design results by providing a range of possible answers or a confidence interval to decision makers and stakeholders, 
with all the relevant assumptions and modeling limitations clearly documented, rather than simply presenting single 
valued answers. The goal of uncertainty management should be to provide to the decision maker as much visibility 
into the limitations and risks inherent in the analysis or design process as possible, so that he or she would come to 
the same technical conclusions as the analyst, given all the relevant data and assumptions. This moves the decision 
risk from the analyst to the decision maker or stakeholder 42 . It is significant to note that in some cases, if uncertainty 
information can be obtained, and is not presented, that the company, decision maker, and/or the analyst might be 
charged with the equivalent of negligence. Therefore it is incumbent upon decision makers to request uncertainty 
information for results presented to them. 

1. Uncertainty Management in Multidisciplinary Design Optimization 

For the purposes of this discussion, Multidisciplinary Design Optimization (MDO) 43 " 44 refers to the part of the 
total design process that can be formulated as an optimization problem. MDO problems usually start out as 
collections of autonomous disciplinary analyses with diverse data formats and diverse levels of fidelity and 
uncertainty in the models that describe the constituent disciplines. The autonomy, complexity, and diversity present 
a major challenge for the integration of disciplines into a nonlinear programming problem statement, quantifying 
various types of uncertainty associated with the MDO models and process, propagating the uncertainty and 
managing it in the design process. We describe several efforts aimed at managing uncertainty in multidisciplinary 
design. 

The development of analytically founded approaches to MDO problem formulation is important in deterministic 
MDO because in realistic MDO environments, it is often difficult to determine a priori whether a chosen problem 
formulation will produce satisfactory results. Reconfiguration of the problem may be required, but the expense and 
complexity of integration usually leaves no room for experimentation with alternative formulations. The difficulty is 
exacerbated with the need to manage uncertainty, because the presence of uncertainty always increases 
computational expense. This has motivated an ongoing effort in building a systematic analytical foundation for 
MDO problem synthesis and solution. 

The effort was first focused on the analysis of existing MDO formulations. The analysis and the attendant 
numerical demonstrations revealed the direct influence of MDO problem formulation on computational tractability 
of the resulting optimization problem 45 ' 48 . The analysis clarified why and how all MDO problem formulations are 
related to each other through appropriate elimination of variables and constraints. 

The basic relationship among all problem formulations and the fundamental difficulty in choosing an appropriate 
formulation a priori point to a clear need for flexible MDO problem implementation that should assist researchers 
and practitioners in formulating and reconfiguring MDO problems with ease, as well as in extracting information 
that would enable reasoning about various aspects of formulations, including the associated uncertainties, and their 
use in the context of analysis and optimization. 

Developing such a general, flexible methodology for managing disciplinary subsystems in MDO problem 
formulation and solution is a feasible undertaking because the context of optimization requires a limited number of 
basic entities to be manipulated: design variables, objective and constraint functions obtained by evaluating the 
outputs of the contributing analyses, and possibly the associated derivatives. Recent developments in computational 
frameworks have eased various implementation aspects for simulation-based design. The mechanics of reasoning 
about MDO problem formulation and implementation, however, have remained elusive, possibly for reasons of 
difficulties in specifying MDO problems. Recent developments 49 ' 50 suggest that the complexity of MDO 
specification can be overcome by a linguistic, context-free, grammar-based approach to MDO problem description, 
formulation, and solution, called reconfigurable multidisciplinary synthesis (REMS). 

REMS is a conceptual framework that comprises an abstract language and a collection of processes that provide 
a means for dynamic reasoning about MDO problems in a range of contexts, with assistance from computer science 
techniques. REMS starts with a description of disciplinary data according to the rules of a grammar. Lexical analysis 
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of the description followed by linkages of multidisciplinary graphs allows the researcher to manipulate basic 
computational components in a number of contexts. The components can be assembled into various MDO problem 
formulations and solution algorithms, including hybrid strategies, with relative ease. The range of contexts for 
reasoning about MDO spans tasks from error checking and derivative computation to formulation and reformulation 
of optimization problem statements. Computational implementation of REMS is in progress. 

Rigorous analysis of MDO problem formulations enables further consideration of uncertainty in MDO problems. 
SACD has supported the efforts of a group at Vanderbilt University in developing methods for managing 
uncertainty in MDO. The effort has resulted in a number of systematic approaches to reliability-based MDO and 
other methods for designing systems in the presence of uncertainty 51 ' 55 . 

E. Robust Design Methods 

1. Approximation and Model Management Optimization 

First-order approximation/model management optimization 56 ' 60 (AMMO) is a rigorous methodology for 
solving high-fidelity design optimization problems with minimal expense in high-fidelity function and derivative 
evaluation. AMMO combines the traditional engineering practice of using models of varying fidelity with state-of- 
the-art nonlinear programming algorithms. It is a general approach applicable to any derivative based optimization 
algorithm and any combination of high-fidelity and low-fidelity simulations, e.g., the solution of the governing 
differential equations on meshes of varying degree of refinement (variable-resolution) or the use of a range of 
physics, from detailed to less accurate (variable-fidelity physics models). 

In proof-of-concept demonstrations, basic AMMO algorithms have been shown to yield from three to seven- 
fold savings in terms of high-fidelity simulations, while attaining high-fidelity optima. The work in AMMO quickly 
spread worldwide, initiating related lines of research in universities, national laboratories, and industry and leading 
to improving or enabling tractability of single-discipline and multidisciplinary design problems. AMMO algorithms 
are now studied as a candidate for reducing the cost of uncertainty-based design. 

The idea of AMMO is to transfer the computational load from high-fidelity models alone to low-fidelity 
models corrected with high-fidelity information occasionally but systematically. If the corrected lower-fidelity 
model does not predict the trends of the higher-fidelity model adequately, AMMO can resort to a higher-fidelity 
model in the available suite of models or it can take shorter low-fidelity steps before requiring high-fidelity re- 
calibration. In the worst case, AMMO is conventional optimization with the high-fidelity model. 

The use of variable-fidelity physics models presented the most intriguing demonstration. In this case, although 
one may expect similar global trends in some regions of the design space, there is no guarantee of similar trends, in 
general. For instance, in aerodynamics, when viscous and shear effects become active, the trends of low-fidelity 
models that do not capture such effects may differ from those of the high-fidelity models. 

One of the proof-of-concept tests considered an aerodynamic optimization of a multi-element airfoil designed 
to operate in transonic conditions 58 . The transonic free-stream Mach number and the multi-element nature of the 
airfoil make inclusion of the viscous effects very important for obtaining physically correct results. This is 
confirmed in Fig. 21, which depicts the Mach number in the flow for the high-fidelity and low-fidelity models, 
where the boundary and shear layers are clearly visible in the viscous case. To capture the viscous effects, the 
governing equations of the high-fidelity model are the Reynolds-averaged Navier-Stokes (RANS) equations. The 
low- fidelity model is represented by the Euler equations. The analysis in both the RANS and Euler modes is 
provided by FUN2D 61 , an unstructured mesh flow solver that also provides the sensitivity derivatives via the adjoint 
approach. The mesh for the viscous model consists of 10449 nodes and 20900 triangles. The mesh for the inviscid 
model comprises 1947 nodes and 3896 triangles. The free-stream Mach number is M* = 0.75, the Reynolds number 
is Re = 9x 10 6 , and the global angle of attack is one degree. 

Figure 22 depicts the level sets of the drag coefficient for the viscous and inviscid models. The solution for 
each problem is marked with a circle. The problem manifests the most adverse situation: not only is the low-fidelity 
model not a good approximation of the high-fidelity model but, in fact, the descent trends in the two models are 
reversed. 

The experiments were conducted on an SGI™ Origin™ 2000 workstation with four MIPS RISC R10000 
processors. Time per one low-fidelity analysis was approximately 23 CPU seconds with attendant sensitivity taking 
between 70 and 100 seconds. One high-fidelity analysis took approximately 21 minutes with 21 to 42 minutes per 
sensitivity. 

The problem with two variables was first solved with the high-fidelity model alone using a commercial solver, 
in order to obtain a baseline number of analyses to find an optimum. The problems were then solved with AMMO. 
As the time per low-fidelity computation was negligible in comparison to the high-fidelity computation, the savings 
were estimated strictly in terms of high-fidelity evaluations. 
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Because the ability to handle high-dimensional problems is an important attribute of first-order, the 
experiments were repeated for the problem with 84 variables. The additional variables were represented by 
parameters describing the shape of the airfoil. The savings were similar to those of the two-variable case: in both 
cases, AMMO yielded five-fold savings in terms of high-fidelity analyses. 

The analysis of the results revealed that the savings were not surprising despite the dissimilarity of the model 
trends. This is illustrated in Fig. 23 for the two-variable case. The plot on the left shows the level sets of the high- 
fidelity model with the solution. The plot on the right depicts the level sets of the low-fidelity model corrected at the 
initial point. The initial point is marked by a square. The correction is applied to the entire region to visualize its 
effect: the correction, containing both the function and derivative information at the initial design, reversed the trend 
of the low-fidelity model, allowing the optimizer to find the next iterate in the northwest corner of the plot, marked 
by a circle. AMMO located the solution (a = 1.6305°, flap y-displacement=-0.0048) of the high-fidelity problem at 

the next iteration. The high-fidelity drag coefficient at the initial point, C"" ,,a/ =0.0171 and at the solution, 

Cl‘ naI = 0.0148 , yielded a decrease of approximately 13.45%. The drag reduction of the 84-variable case was 
approximately 25%. Details of the demonstration may be found in Ref. 58. 

The relation of AMMO to managing uncertainty in design is two-fold. First, AMMO implicitly reduces 
uncertainty associated with design by enabling the use of higher-fidelity modeling at decreased cost. Second, 
AMMO-based ideas can be used in decreasing the cost of explicit uncertainty-based design optimization 62 . 



Figure 21 - Mach contours for transonic flow around an airfoil, high- and low-fidelity models. 
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Figure 22 - Contours of objective function for transonic flow 
around an airfoil, high- and low-fidelity models. 
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Figure 23 - Illustration of design under uncertainty using 
AMMO optimization using high- and low-fidelity models. 

2. Design Under Uncertainty in the Flight Optimization System (FLOPS) Code 

As described in Ref. 12, the method of moments was implemented in the FLOPS aircraft mission analysis 
program for several classes of statistically independent, normally-distributed, random input variables and several 
classes of random output variables, to enable robust design. Different levels of input uncertainty and required 
constraint satisfaction were imposed. The effect of uncertainty on the design point, compared with a deterministic 
design, is noted. In Ref. 12, the example was also analyzed and validated with a Monte Carlo simulation; additional 
work was performed using the simulation-search method within the UNIPASS tool. 

In this study, the effects of uncertainty in two input aircraft design variables are considered for the purposes of 
illustrating uncertainty propagation through the FLOPS code and robust design under uncertainty. For the 
demonstration (and following the derivation in Ref. 11), these input variables were assumed statistically 
independent, random, and normally distributed about a mean value. These assumptions simplify the implementation 
and help quantify the input uncertainties. The assumption of the variables being statistically independent is not 
required; correlation between the variables can be easily accounted for within the formulation at the cost of more 
computational work. For non-normal input distributions, the methods of moments’ corrections are only 
approximately correct 

For simplicity, the demonstration was derived from a particular sample cases distributed with the FLOPS code, 
which is the of five-design variable subsonic transport design (xfp2.in). The input file was modified to allow only 
the variables THRUST (the maximum rated thrust per engine, in pounds force), and SW (the wing reference area, in 
square feet), to be active design variables; upper and lower bounds were also specified for these design variables. 
The modified input file was then used for both deterministic and robust optimizations. The robust optimizations had 
various levels of input uncertainty for the active design variables and various levels of required constraint 
satisfaction, both specified in auxiliary input file to FLOPS. For the example case, the specified input variable 
uncertainty corresponds to 5% of the mean value for each of the two input variables. Since the mean value of 
THRUST for this problem is 47,500 lb and the mean value of SW for this problem is 2272 ft 2 , the variability 
corresponds to 2375 lb and 113.6 ft 2 for the two variables, respectively. The constraint satisfaction requirement 
ranged from zero (where the constraint was satisfied with 50% probability for a normal distribution) to three 
(constraint satisfied with 99.9% probability for a normal distribution). The optimization objective was specified to 
be the aircraft gross takeoff weight. The seven possible aircraft performance constraints, normally activated with this 
sample problem, were used. These included the aircraft required range (which were held fixed for this problem), the 
approach speed, the takeoff and landing distances, the approximate missed-approach and second-segment climb 
gradients, and the excess fuel. The Broyden-Fletcher-Goldfarb-Shano (BFGS) optimization method (the default 
among several optimization methods available within the FLOPS code) was used to solve this problem. In the 
FLOPS implementation of this optimization method, a composite objective function was minimized. The composite 
objective function was composed of the true objective augmented with a highly nonlinear penalty function that grew 
rapidly as the design variables approach their upper or lower bounds, and as constraints become active. 

Figure 24 is a simplified aircraft sizing contour ("thumbprint") plot illustrating the design space near the 
deterministic and robust optimizations for this case. The x-axis of the figure shows a normalized value of the 
maximum rated thrust per engine, ranging from 20,000 to 60,000 pounds force; the y-axis of the figure shows the 
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normalized value of the reference wing area, ranging from 1000 to 5000 ft 2 . The figure is simplified from a typical 
thumbprint plot in that only the active constraint violation boundaries for the specified problem are shown, rather 
than multiple contours representing various values of the limit state function. The constraint violation boundaries 
were interpolated from a parametric variation of the two variables with nine equally spaced points in each direction. 
Also shown in Fig. 24 are the locations of final design points for the deterministic and three mean robust 
optimizations. In this example, both FLOPS constraint 2 (the upper limit on approach speed) and FLOPS constraint 
5 (the lower limit on missed approach climb gradient) were active for the deterministic optimization. As expected, 
the deterministic design point is found at the intersection of the two active constraints. The optimization path was 
almost entirely within the feasible region in the figure. 



Figure 24 - Aircraft thumbprint plot illustrating 
robust design of a subsonic transport jet. 


The three robust optimization points (labeled K = 
1, 2, and 3 in the figure) correspond to imposed 
constraint satisfaction margins of 1, 2, and 3 standard 
deviations about the mean value of the deterministic 
solution. The offset in the robust design points from the 
constraint violation boundaries is proportional to both 
the imposed input uncertainty and the gradient of the 
constraint with respect to the uncertain design variables. 
Fig. 24 also shows that the robust optimization with K = 
1 enforces a greater margin of satisfaction for both 
constraints than does the deterministic optimization. 
Similarly, each of the robust solutions enforces greater 
constraint satisfaction, with increasing values of K than 
either the deterministic solution or the robust 
optimizations with smaller values of K. For the 
deterministic optimization, the constraint satisfaction 
with respect to single constraint violation is only 50% 
probability for an output normal distribution; for K = 1, 
this probability increases to about 84%; for K = 2, the 
probability is about 97.7%; and for K = 3, the probability 
rises to about 99.9%. 


Simultaneously (not shown in the plot), the aircraft weight increases with increased constraint satisfaction from 
the deterministic value of 213110.5 lb to a value of 217052.6 lb for K = 1, to a value of 220222.0 lb for K = 2, and 
to a value of 223443.1 lb. for K = 3. The weight for K = 3 is about 5% higher than the deterministic solution. 

3. Design Under Uncertainty for Coupled Aero-Elastic Wing 

A system has been developed to couple an iterative Computational Fluid Dynamics (CFD) solver and a Finite 
Element Method (FEM) structural solver with gradient-based optimizers, First Order Second Moment (FOSM) 
uncertainty analysis, and First Order Reliability Method (FORM) analysis 63 " 64 . The system has been applied to 
perform deterministic optimization, robust optimization, and reliability analysis for a flexible trapezoidal wing. The 
unique feature of the system is that the iterative loops of the methods (Euler/Navier-Stokes CFD, coupled 
aero/structural interaction, quasi-analytic sensitivity analysis, Newton method-based optimization. Most Probable 
Point (MPP) reliability analysis) are solved simultaneously. That is, a well-converged solution to any inner iteration 
loop is not required - or even expected - until the outermost iterative loop reaches convergence. This feature has 
been shown to significantly reduce the computational expense relative to simply nested analyses. Previous 
papers 63 " 65 documenting this work have compared the method to other Simultaneous Analysis and Design (SAND) 
methods. An important property of this method is that rather than performing all the functions in a monolithic code 
that would need to be validated, SASDO allows us to assemble familiar analysis tools for each part of the analysis. 
Quasi-analytic sensitivity analysis was enabled for the major analysis components by automatic differentiation of 
the source code. Several of the interface methods were easily differentiated “by hand’’. 

The system was applied to a simple trapezoidal planform wing of dimensions similar to a commercial transport, 
flying at transonic speeds typical of a commercial transport. The CFD and FEM analysis methods are described 
more fully in the referenced papers and their references. The FE structure model was composed of spars, ribs, and 
skin as shown in Figure 25. The aerodynamic characteristics were modeled by a finite volume CFD solver, CFL3D. 
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The mesh on the wing surface is shown in Fig. 26 after it 
has been deflected due to aerodynamic loads. Structural 
element sizing parameters and geometry parameters such as 
wingspan, airfoil maximum thickness, and airfoil camber 
were used as design variables in several deterministic 
optimizations. In order to reduce the number of parameters, 
the structural element sizing parameters were grouped 
geometrically such that one parameter attributed to each zone 
shown in Fig. 25 was used to control the sizes of all the 
elements within the zone. Uncertainty in several of those 
variables was propagated through the coupled analyses to 
conduct robust optimizations and reliability analyses. For 
optimization studies the objective function was typically the 
ratio of lift to drag. Constraints were imposed on geometric 
quantities such as leading edge radius, on discipline-specific 
quantities such as tip deflection, and on quantities using 
information from multiple disciplines, e.g. compliance, 
which is the work done by the aerodynamic loads to deflect 
the structure, and payload, which is the difference between 
the aerodynamic lift force and the structural weight. Those 
were also the dependent variables in the robust design and 
reliability analysis studies. 

A comparison of some robust design results is shown in 
Figs. 27-29. The root airfoil thickness and camber, and the 
element thicknesses in the two inboard zones were 
considered as uncertain design parameters. A coefficient of 
uncertainty of 0.0001 was attributed to each. The 
deterministic optimization was performed first. Then several 
robust optimizations were performed using the deterministic 
result as the initial design. The optimizations were conducted 
accounting for several levels of uncertainty. The uncertainty 
effects on the optimization process are most notable in the 
way the constraints are satisfied as shown in Figure 27. For 
robust design using a FOSM method, the deterministic 
inequality constraint, g < 0 , is met with some desired 

probability by representing it as g+k<7 < 0 , where g is the 

mean value of the constraint, G g is the standard deviation of 
the constraint determined by propagating the uncertainties in 
the uncertain variables, and the parameter k determines the 
probability of satisfying the constraint. An increase of the 
parameter k indicates an increase in the specified target 
probability of satisfying the constraints. Assuming a normal 
Gaussian distribution of the output variables, values of k = 1, 
2, and 3 would represent probabilities of 84.13%, 97.73% 
and 99.87%, respectively. In Fig. 27, the bars show the mean 
Figure 27. Effect of uncertainty on constraints. value of the constraint functions, while the circles show the 
offset required to meet the constraint with the desired probability. The three constraints shown are the difference 
between the aerodynamic lift force and the structural weight, the compliance, and the pitching moment coefficient. 
The constraints as shown in the figure have been scaled. Figure 28 shows the changes in the design variables, the 
mean values of the parameters, resulting from the deterministic and robust optimizations. The optimizer tends to 
reduce the root thickness to decrease the shock strength, thereby reducing the drag and increasing the ratio of lift to 
drag until the constraints become active. Increased target probability of satisfying the payload constraint requires the 
structural weight to decrease and/or the lift to increase. Reduced element sizes reduce the weight but allow increased 


Figure 26. CFD solution (pressure contours) 
shown on deflected surface mesh. 
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Figure 28. Effect of uncertainty on 
design variables. 



Figure 29. Effect of uncertainty on 
objective function. 


bending and twist. The twist is exhibited in the form of washout, which helps to reduce the pitching moment and 
increase the probability of meeting that constraint, but it also decreases the lift. The root airfoil camber is increased 
to offset the reduced lift. The optimizer balances these conflicting tendencies. Figure 29 shows that, in the 
neighborhood of these design values, the objective function, the ratio of lift to drag, is relatively insensitive to the 
design variable changes necessary to satisfy the constraints. 


F. Reliability Analysis Methods 

An unconventional application of reliability methods is now described; more conventional reliability 
assessments are discussed in Ref. 6. Reliability assessments were performed for the results of the aero-elastic robust 
designs (described previously in section E3) using three FORM approaches: the Flasover-Lind-Rackwitz-Fiessler 
(F1L-RF) Reliabilty Index approach (RIA), the Performance Measure Approach (PMA) and a PMA-based RIA 
(PRIA) 65 . A fundamental difference between the FOSM robust design assessment of uncertainty and the FORM 
reliability assessment of uncertainty is where in the design space the functions are evaluated. The uncertainty for 
robust design is characterized by information evaluated at the mean value of the output function; whereas, the 
uncertainty for reliability assessment is characterized by determination of a Most Probable Point (MPP) and 
evaluation of the constraint function at that point. The MPP can be determined by an optimization process. The RIA 
approaches produce a reliability index (3 that, similarly to the parameter k used in the robust design process, 
corresponds to a probability of satisfying the constraint. The performance measure g p is the offset from the 

constraint (limit-state function) that will result in the target reliability. A detailed description of mathematical basis 
and the solution methods for the three approaches are described in Ref. 63 and its associated references. Table 3 
shows a comparison of the reliability assessment parameters and the robust design parameters. For the RIA results, 
the comparison can be drawn between the reliability index (3 and the target k specified in the robust design. For the 
PMA results the comparison can be drawn between the performance measure g and the target constraint value 
g + ka y used for the robust design. The reliability analyses are consistent with each other; but, the agreement with 

the robust design seems to degrade with increasing values of the robust design target probability P Differences in 
the results of the two methods can come from several sources. The uncertainty for both the robust design method 
and the Reliability Assessment methods is characterized using only first order information. Although the input 
parameter uncertainty is characterized by a Normal distribution, for a non-linear function such as those resulting 
from the coupled aero/structural analysis, there is no expectation that the output distribution is also Normal; but, the 
parameters being compared are derived on the assumption that it is. As described earlier, the uncertainty information 
for the robust design is evaluated at the mean value, which more accurately characterizes the center portion of the 
distribution, whereas, the uncertainty information for the reliability assessment is evaluated at the MPP, which more 
accurately characterizes the tails of the distribution. In addition to the mathematical basis for differences, the 
processes for the robust design and the reliability assessment are iterative methods, as are the underlying 
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multidisciplinary analysis methods, so convergence tolerances for those iteration loops have potential for error. But 
the differences for all the cases shown are relatively small. 


Table 3. Comparison of Reliability Assessments of Robust Designs. 


Constraint 

PMA 

Robust Design 

Reliability Index Approach 

HL-RF 

PRIA 

* 

g + ka H 

k 

P f 

P 

i-Pf 

P 

t-Pf 

g(L-W) 

-0.00219 

0.00005 

i 

0.8413 

1.206 

0.8862 

1.199 

0.8848 

g(L-W) 

0.00122 

0.00031 

2 

0.9773 

1.879 

0.9699 

1.859 

0.9685 

g(L-W) 

0.00017 

-0.00075 

3 

0.9987 

2.983 

0.9986 

2.939 

0.9984 

g(C m ) 

-0.00019 

-0.00003 

1 

0.8413 

1.038 

0.8504 

1.050 

0.8531 

g(C m ) 

-0.00873 

-0.00500 

2* 

0.9773 

3.471 

0.9997 

3.586 

0.9998 

g(c m ) 

-0.00695 

0.00045 

3 

0.9987 

4.164 

1.0000 

4.165 

1.0000 


* Constraint active, but not tight 


III. Risk Analysis / Management 

This past year, FY2006, SACD worked with NASA’s Exploration Systems Mission Directorate, (ESMD) to 
define how this Directorate will utilize the concept of Continuous Risk Management (CRM) 2 to assist in managing 
the wealth of activities which must occur to fulfill the nation’s space exploration vision. This “Vision for Space 
Exploration” (http://www.nasa.gov/pdf/55583main_vision_space_exploration2.pdf) is broad, dynamic and spans the 
technological gamut from utilization of State-of-the-Art technology, to future integration of technologies which 
currently may only be laboratory investigations. The Vision provides for the milestones of replacing NASA’s 
current space shuttle system with a Crew Exploration Vehicle (CEV) which initially has docking access to the 
International Space Station (ISS). A launch vehicle, Crew Launch Vehicle (CLV) is also being designed to place 
the CEV into Earth Orbit. Heavier cargo is going to be delivered by another launch vehicle, the Cargo Launch 
Vehicle, (CaLV). As confidence in the CEV system grows, mission goals will start to include requirements for 
Lunar surface access and science and technology demonstration activities. Additional flight elements such as an 
Earth Departure Stage, Robotic Elements, and a Lunar Surface Access Module, will be required. Again as 
confidence in NASA’s ability to function in a lunar environment grows, mission goals will move on and push for the 
same type of exploration activities to occur on the Martian surface. Beyond the activities envisioned for Mars and 
depending upon the success of the Vision’s objectives, activities which reach ever further into the space 
environment may be pursued. 

The long term success of NASA’s exploration initiative also requires that it be performed in a safe, productive, 
and cost controlled manner. To formalize the Safety and Program Management activities which must be integrated 
across all of the above activities NASA has chosen to implement CRM with tight control via utilization of the 
commercially available Risk Management computer program, Active Risk Manager (ARM) 66 . For a project of this 
magnitude CRM is mandated by NASA Policy Regulations “NASA Program and Project Management Processes 
and Requirements” 1 , and “Risk Management Procedural Requirements” 2 . Active Risk Manager brings to the table a 
process tailorable computerized database for Risk Management. Client access to the database requires only a 
personal computer with web access and an account in NASA’s single sign on collaborative engineering environment 
ICE, Integrated Collaborative Environment. NASA’s CRM process is mapped to the ARM application to provide 
all users in ESMD and its sub-tier organizations, including contracting organizations, a consistent interface to risk 
data no matter which parent organization is being directly supported. Tight control in this case implies that certain 
levels of management, Directorate/Program/Project, each incorporate ARM into their Risk Management process. 
Certain report formats available to all ARM users can be used to report RM information in a consistent manner 
throughout the organizational hierarchy. A standardized “5x5” Risk Classification matrix is prescribed and the three 
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levels of impact, low, moderate and high are mapped to this classification matrix. RM tool and reporting 
requirements and other “Tight Control” procedures do not however try to dictate to such a fine grain that RM 
information becomes un-informative at the lowest organizational tier level where risk management is actually 
occurring. Risk managers are free to tailor terms in the definition of their risks such that they are handled by 
appropriate resources. ESMD recognizes that effective risk management occurs when mitigation activities are 
delegated to the organization element closest to the problem. Risks at lower levels but of great consequence are 
reported up to higher level organizations and the ARM database facilitates this process. 

Effective Risk Management across a multi- 
organizational hierarchy is dependent upon efficient and 
unfettered access to data by all stakeholders. ARM provides 
for user definition of a hierarchical structure of risk data. In 
ESMD’s case, as illustrated in Fig. 30, this hierarchy is 
based upon both the Organizational Breakdown Structure 
(OBS) and within organizations the Work Breakdown 
Structure of particular tasks (WBS). ARM access is 
provided on a read-only basis to practically any person 
involved in supporting ESMD’s goals and objectives. A 
typical user can see all risk data within the database, not just 
data within their parent organization. This physically 
facilitates the concept of Risk Communication and fosters 
an attitude of collaboration, as discussed in Ref. 7 as a 
desirable attribute of corporate culture for NASA to nurture. 

Concurrent with the rollout of ESMD’s Risk 
Management and incorporated in that organization’s Risk 
Management Plan, is the inclusion of Knowledge 
Management activities. NASA has historical data from 
several programs where knowledge and “Lessons Learned” 
have been formally gathered. The ARM data structure is 
Figure 30. ESMD Risk Breakdown Structure also being utilized as a repository for consolidation of this 

data. Providing quick access to lessons learned information 
is intended to assist other ESMD-ARM users who may be wondering what type of risks their program may incur, 
and may provide historical mitigation success and failure data. To further facilitate communication between 
organizationally and geographically disperse ESMD team members, Communities of Practice (CoP) have been or 
are being set up in the following areas: 

• Risk & Knowledge Management 

• Probabilistic Risk Assessment 

• Strategic Analysis 

• Earned Value Management 

• T echnology Protection 

• Systems Engineering & Integration 

• Research and Technology 

Focus for these CoPs are on activities related to ESMD’s objectives, but it is also a web location where more 
general discipline dependent information can be shared. Another Knowledge Management activity encouraged by 
ESMD is the use of “Pause and Learn” (PAL) events. A PAL is modeled after the Army process entitled “After 
Action Review”. At the end of selected milestones along the life -cycle of project activities, recent participants pause 
and have a meeting to discuss what is going right and what is going wrong with what has gotten them to the state 
they are currently in. The meeting is chaired by an independent facilitator and participants are free to bring up all 
issues for discussion and logging into meeting minutes. Such reflection hopes to celebrate what is currently going 
right on a project and alert participants to an agreed upon area which may require a change in approach to manage 
properly. 

One forward looking point which NASA will emphasize in future upgrades to its ESMD Risk Management 
procedures is the incorporation of “Opportunity” as well as the traditional “Threat” type of Risk data. The Vision for 
Space Exploration can and should inspire new technologies to more efficiently meet its goals as time progresses. 
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G ft Chief Engineer 
G ft Directorate Integration Office 
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The Risk Management process can formally track Opportunities as it currently does Threats, and communicate that 
information across the dispersed program management hierarchy. This additional exposure of technologies to 
programs and program activities to technology developers is envisioned to create a working level synergism 
between the activities of these two groups and their managers. 

Finally an effective RM activity has to integrate with other project data such as cost/schedule/resource (project 
management) data, and product requirements, concepts of operations and element functionality (systems 
engineering) activities. Within the ICE environment will be the Systems Engineering computer program, 
CRADLE 67 , and the Project Management computer program, Primavera 68 . Task requirements in Primavera are 
envisioned to become part of the Risk Activity breakdown data hierarchy in ARM. Cost and schedule impacts from 
Risk Management activities will be linked to the ESMD Primavera database enabling managers to assess 
programmatic risk impacts. Earned Value data for a project can now also include Risk data and give managers 
additional quantified decision making power. If project management (Primavera) data is linked to the ARM 
installation. Risk Managers can utilize the ARM uncertainty analysis feature to assess the variability of cost and 
schedule parameters for risk activities in terms of their impact on overall program planned cost and schedule 
allocations. 


A. Risk Classification: 
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ESMD Risk Classification Matrix 


Reference 2 specifies in accordance with CRM 
techniques that a Risk Classification system be invoked. 
This system shall provide a means for analyzing a risk in 
the sense that the Probability of a risk’s occurrence is 
rated, as is also the risk’s Impact if it were in fact to 
occur. ESMD has standardized the Probability-Impact 
Risk Classification Matrix as shown in Figure 31. Low 
(green), Medium (yellow), and High (red) risk severity 
areas are indicated in the figure. In order to place a risk 
in this classification system a “scoring scheme” is 
required to assist the risk analyst with mapping the 
severity of the risks impact and its likelihood to the Risk 
Classification Matrix. 

ESMD has prescribed a template Scoring Scheme at 
the Directorate Level of the organization. This scheme 
defines four Impact Categories for which a risk may be 
scored. The Impact Categories are Safety, Performance, Cost and Schedule. Either qualitative or quantitative 
criteria may be used to score a risk and a Risk Scorecard is typically defined for each organizational level which 
defines that groups criteria for rating a risk against each of the four impact categories. Figure 32 shows the “Score 
Card” proposed for use by ESMD’s Crew Exploration Vehicle (CEV) project. 

Further risk analysis is prescribed in Ref. 2 in that the risks identified should be prioritized in a formal “Risk 
List”. ARM will in automated fashion generate these Risk Lists based upon the Risk Scores assigned to each risk by 
virtue of its location in the Risk Classification Matrix. However a program manager typically desires greater control 
over this prioritization, and may desire to remove the ambiguity which will occur in a large program where multiple 
risks assigned to the same risk matrix cell will have the same Risk Score. Program Manager prioritization and 
reporting of a Risk’s Priority along with its Score is a report which should be used to show this effect. Note also that 
the Risk Scores assigned by classification to a cell in Fig. 31 are not equal to a multiplication of the ordinal values 
for Probability and Impact. Rather the cell scores are assigned based upon a logic which in this implementation 
biases higher scores towards risk which are more affecting by their consequence than by their probability ranking. 
This scoring scheme represents a slight divergence over what NASA has used for other typical Risk Management 
activities 69 " 71 however it more clearly emphasizes the issue of biasing consequence as having a slightly greater effect 
on risk score than likelihood. 
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Figure 32 - CEV Scoring Scheme, A typical project level Scorecard 


B. Risk Management Process and Options: 

The ESMD Risk Management Process is structured from a “corporate model” where responsibility is held at the 
lowest possible effective organizational levels. Top level direction and distribution of resources flows down to the 
sub-tier organizations; so too does the Risk Management process. At the agency level, Refs. 1 and 2 prescribe Risk 
Management requirements for ESMD. At the next level the Directorate has created a Risk Management Plan where 
it prescribes the necessary process requirements to ensure proper sharing of data. Reporting and tool integration 
consistencies are part of these requirements. Below the Directorate level the Program level organizations create 
individual Risk Management Plans which suit their program management requirements and still adhere to the higher 
level requirements. Similarly the Project level organizations create a Risk Management Plan which includes 
Program, Directorate, and Agency prescriptions. For large programs/projects a formal risk review board may be 
configured and meet regularly to adjudicate risks. For smaller or less disparate activities the Risk Management 
activities may occur formally only as a part of regular program/project management meetings. Each organization in 
ESMD defines a Risk Management Officer (RMO) who may or may not be a person distinct from that element’s 
Program/Project Manager. The RMO is responsible for assisting the PM and implementing the organizations CRM 
process. The RMO is also responsible for ensuring CRM/ARM training for personnel in his organization is supplied. 
Directorate level training is provided to all, but more specific functionalities and process requirements may be 
enforced at the lower organizational levels. 

One of the most useful things a program manager can do to effectively utilize the CRM process is to track 
Technical Performance Measures, Key Performance Parameters, Figures of Merit and other such program 
measuring-sticks as risks. The International Council on Systems Engineering describes the advantages gained by 
keeping such data in a Risk Management system 72 . 

As previously highlighted, Communication is a primary asset of CRM. For ESMD’s CRM/ ARM installation, 
communication of risk data to appropriate managerial levels is handled in three particular manners. First the 
database structure and access privileges as we have discussed promote sharing of risk data across the entire 
Directorate, its sub-tier organizations, and its supporting contracted service vendors. Secondly the reporting 
capability and a feature called “Alerts” enables users to use the ARM tool along with their Risk Review procedures 
to show risk data to other organizations either horizontally or vertically throughout the organization structure. Such 
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alerts can be used for informational purposes as well as to document the needs for requesting additional risk 
mitigation resources. Finally if a particular organization desires that a risk they are in charge of be moved to another 
more appropriate responsible organizational element, this can be accomplished by a risk transfer. A risk transfer to a 
parent organizational element may be termed a Risk Escalation. 

C. Decision Methods 

Successful Continuous Risk Management requires sound decision making for risk identification, analysis, and 
control. The Logic-Evolved Decision (LED) analysis software framework is a modeling and decision support tool 
for performing risk-based analyses, assessments, and prioritizations for a broad range of applications. This top-down 
analysis approach uses software tools and accompanying methodology developed at the Los Alamos National 
Laboratory (LANL) for modeling the behavior of complex systems with respect to decision support applications. 
The LED computational algorithm uses linked, formal logic models to represent the basic functions of a decision 
analysis tool. This systems analysis capability incorporates fuzzy logic, approximate reasoning, possibility, 
probability, multi-attribute scoring, and graph theory to construct these decision support models. LED is a flexible, 
self-contained, comprehensive, and traceable decision support software tool for risk-based prioritization, hazard 
assessment, and portfolio management across a broad spectrum of applications. 

The use of LED techniques has proven scalable, flexible, and reproducible for a wide range of decision support 
applications. The scalability of the LED analysis technique allows for detailed analysis of specific problems as well 
as a global tool for risk-based management of major and/or multiple systems. These techniques have been used in 
nuclear weapons safety applications, reducing espionage risk, evaluating sabotage risk, determining the risk of 
protected material theft, risk-based asset and infrastructure management, and evaluating risk-reduction measures for 
natural hazards including lightning and wildfire. The LED analysis approach and methodology has been thoroughly 
peer-reviewed by independent organizations including the American Society of Mechanical Engineers and the 
Monterey Institute for International Studies. The Monterey Institute’s extensive review concluded that the LED 
approach is the most comprehensive counter-terrorism risk tool available. 

The discussion in the Decision Methods section of this paper focuses on the collaborative effort between NASA, 
LANL, the National Institute of Aerospace, and Volpe Center Department of Transportation to develop risk-based 
assessments for the purpose of improving the overall security of the national Air Transportation System (ATS). The 
objective of these assessments is to provide a decision support tool that can be used to prioritize NASA research in 
aviation security based upon a comprehensive ATS risk assessment and other portfolio management attributes 
pertaining to technology development and implementation. 

1. NASA ’s Application of the LED Process: Risk-based Prioritization of NASA Research in Aviation Security 

Recently, LED techniques were used to identify, assess, and rank order the risks facing the United States ATS. 
An analysis of the threats facing the entire spectrum of aviation activities within the ATS was conducted in an effort 
to prioritize the NASA Aviation Safety and Security Program (AvSSP) security research portfolio. This LED 
portfolio prioritization process developed for the NASA AvSSP was based upon a comprehensive ATS risk 
assessment. Although many metrics are available for the technology prioritization process, such as cost, technical 
development issues, etc., the initial metric chosen for the technology rank ordering was the risk reduction associated 
with terrorist attacks on the ATS 73 . This top-down analysis effort assisted NASA in prioritizing its research in 
aviation security by providing NASA with a risk-based rank ordering of technology impact on national ATS terrorist 
attacks. As a result, this analysis and assessment effort yielded a two-fold purpose by accomplishing two objectives 
of national importance: 

• A comprehensive ATS risk assessment based on terrorist attacks was developed for the nation. 

• A decision support tool for the purposes of prioritizing NASA research in aviation security was 
developed. 

For the purposes of this analysis, the concepts of combat aircraft survivability are used to examine terrorist 
risk 74 . An attack scenario is a complete description of the process by which an adversary carries out an operation 
against a target. Risk is the expected loss and terrorist risk is evaluated for individual attack scenarios in this 
analysis. Total risk is determined from an aggregation over a set of scenarios for one or more targets of interest. Risk 
is expressed in terms of susceptibility, vulnerability, and consequences by the following relationship: 
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Susceptibility ® Vulnerability ® Consequences ■=> Risk 
where 

Susceptibility ® Vulnerability Likelihood 
of an attack scenario. 

Translated to the problem being studied, security of the ATS, and linguistic expressions, susceptibility 
corresponds to the likelihood that an attack scenario is completed. Furthermore, susceptibility depends upon (1) the 
likelihood that the attacker chooses to attempt the attack (attractiveness), (2) the likelihood that the attack is carried 
to completion absent any intervention, and (3) the likelihood that the defender can prevent the attack from being 
completed. Vulnerability is the likelihood that the level of damage (the consequences given completion of the 
scenario) is such that the attack is considered to be successful by the attacker. Vulnerability depends upon (1) the 
intrinsic response of the target to the attack and (2) the capability of the defender to mitigate the consequences of the 
attack. Likelihood is thus a measure of the “belief’ that an attack scenario will occur. Likelihood can be expressed in 
the following ways: 

• Linguistic. “A is very unlikely” ; “A is more likely than B” 

• Frequentist : f(A) = 2xl0" 3 /year ; f(A) > f(B) 

• Probabilistic. p(A) = .001 ; p(A) > p(B) 

• Odds'. Payoff indifference between choices 

Due to the lack of historical data concerning terrorist attacks against the ATS, the linguistic approach incorporating 
approximate reasoning is chosen to infer the likelihood and consequence of terrorist attacks against the ATS. The 
approximate reasoning approach incorporated in this LED based analysis and assessment uses expert judgment to 
infer the outcome, Likelihood ® Consequences ■=> Risk, of terrorist attacks against the ATS. 

The NASA AvSSP approach is to consider aviation security for the entire ATS. For analysis purposes, the entire 
ATS was therefore systematically divided into 3 distinct sub-systems: (1) aircraft, (2) airports, and (3) airspace. The 
aircraft sub-system was further decomposed into Federal Aviation Regulation (FAR) parts since the risk of a 
terrorist attack on the aircraft sub-system is dependent on the various aircraft operations. The aircraft sub-system 
decomposition included: (1) FAR Part 121 Passenger/Cargo-Outsider and Insider Attacks, (2) FAR Part 121 All 
Cargo, (3) FAR Part 135, and (4) FAR Part 91 aircraft operations. 

After decomposing the ATS into the appropriate sub-systems, the first step in the risk assessment/technology 
prioritization process is to brainstorm the terrorist attack scenarios as shown in Figure 33. This scenario 
development 





A 11 1 




Figure 33 - Brainstorming the Attack Scenario 
Possibilities. 


process involves methodically identifying threats facing the ATS and then determining the attacker/attacker group, 
target selection, planning, logistics, the actual assault, and the ultimate target response for each sub-system of the 
ATS. These elements are then rolled into detailed attack scenarios using a top-down approach for each ATS sub- 
system. Subject matter expert input is used in this attack scenario development step. The various subject matter 
experts used in this step include intelligence analysts, NASA aircraft/aircraft operations experts, aviation system 
expert consultants, pilots, airport managers, air traffic controllers, LANL decision support experts, and NASA 
security project researchers. 
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The next step in the analysis process is to input the detailed attack scenarios for each ATS sub-system into the 
LEDTools software package. The LEDTools software is used to construct the attack scenarios in a hierarchal tree 


Target 


Q The attack targets the aircraft. 

A The attack is . _ 

^ on the airframe, 

| f — 0 The attack originates external to the aircraft. 

^ The attack involves 

I i a weaponry. 

^ The weapon used 

O is a missOe launcher. 

( •] is a man-portable missile. 
V The missile 
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O fails to detonate. 

^ The attacker group consists of 
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Logistics 


Assault 


Target 

Response 



Figure 34 - Hierarchal Tree Structure of an Attack Scenario in LEDTools. 


structure. The LEDTools software contains logical gates and operators that are used to construct the possibility trees 
as shown in Figure 34. Some of the logical gates within LEDTools include AND, Ordered AND, Combination, 
Inclusive OR, various Causal gates, Intersection, Union, Difference, Element, Taxonomy, and Process gates. These 
gates are used in conjunction with textual input to form the attack scenarios in a logical, hierarchal tree structure. 
The possibility trees can be solved in order to obtain a comprehensive, textual set of attack scenarios for each ATS 
sub-system. As a result, the paths of the possibility tree are indeed the attack scenarios. An example of one solution 
to the possibility tree shown in Fig. 34 is displayed below: 


The attack is on the US aviation system. The attack is against the commercial aviation system. The targeted 
system is ciassiTied as a Part 121 air-carrier operation. The air-carrier operation handles passenger and cargo 
traffic. The attack targets the aircraft. The attack is on the airframe. The attack originates externa I to the aircraft. 
The attack involves weaponry. The weapon used is a man-portable missile. The attacker acquires the weapon 
system. The attacker transports the missile system to the attack site. The attacker acquires the target. The 
attacker fires the missile. The missile flies to the target. The missile warhead detonates. The attacker group 
consists of outsiders only. 

This possibility tree solution capability of LEDTools is a very powerful functionality that aids in the attack scenario 
development process since this solution displays each scenario contained within the possibility tree in textual form. 
After all scenarios are developed for all of the possible attacks against the ATS, the scenarios are reduced to ATS 
element spanning sets. This is actually a screening process for developing a workable sub-set of scenarios that are 
representative of a larger class of attacks. This process is helpful since it is easier to deal with hundreds of scenarios 
rather than millions of scenarios. 
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The third step in the analysis process is to develop the risk inferential models for the defender and attacker. The 
risk inferential models include attack Attractiveness of pm suite for an Attack scenario chanae in Risk 

Cosi/Payoff Ratio 

(=>) Change in Risk Associated with New PM Baseline Risk — 

Scenario Risk with Current PM Suite 4 Likelihood ® Consequence 

... ... , _ Expected consequence of successful attack.^ 

I el OO “ • Expected Casualty Consequences Range Of 

Susceptibility^^ # Expected Economic Consequences Conseauences 

® v U I n e ra b i I i ty - E xpected Likelihood of successful attack with current PM S uite. " 

V=?> Scenario attempt likelihood with current PM suite. 


scenario susceptibility, vulnerability, and 
consequence. These models also include 
the baseline risk as it exists today for the 
ATS with the current preventive and 
mitigating measures in place and various 
attacker intent sub-models. The logical 
tree structure of the risk inferential model 
is displayed in Fig. 35 and the various 
computational elements that comprise the 
risk inferential model are shown. The 
approximate reasoning approach used in 
this study utilizes subject matter expert 
elicitation to populate the variables within 
the risk models. The various subject 
matter experts used in this step include 


Target 
Susceptibility 

Target 
Vulnerability 



O Attractiveness of scenario to attacker with current PM suite. 

O Likelihood of agent group formation. 

S\ Scenario success likelihood given attempt with current PM suite. 
i'-j] Defender's estimate of PM likelihood of current PM suite. 

# Defender's estimate of direct PM likelihood with current PM suite. 

Required agent set with current PM suite. 

Required knowledge collection with current PM suite. 

[#1 Defender's estimate of uninhibited success likelihood. 

O Scenario Risk with New PM Suite"^^^^^^^^^^^ Risk with PM 
O New PM Costs _ 

Suite 


Secondary Factors 


Phase 3 Analysis 
Figure 35 - Risk Inferential Model for the Defender and 
Attacker with Call-out for Various Risk Model Components. 


© 


intelligence analysts, NASA 

aircraft/aircraft operations experts, aviation system expert consultants, pilots, airport managers, air traffic 
controllers, LANL decision support experts, nuclear and chemical/biological expertise, electromagnetic effects 
expertise, cost expertise, and NASA security project researchers. 

We are now capable to determine and identify the attack scenario baseline risk for each ATS sub-system which 
includes [the aircraft] FAR Part 121 Passenger/Cargo-Outsider and Insider Attacks, FAR Part 121 All Cargo, FAR 
Part 135, FAR Part 91, the airport, and the airspace. At this point in the analysis process, the first objective for this 
overall systems analysis effort of developing a comprehensive ATS risk assessment based on terrorist attacks for the 
nation is accomplished. An illustrative example of attack scenario baseline risk output for a specific ATS sub- 
system is shown below in Figure 36. 


Dependence of Scenario Risk on Attacker Type 



Scenario 

Figure 36 - Illustrative Example of Attack Scenario Baseline Risk for ATS Sub-system. 
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Step 5 in this analysis process involves mapping and applying the NASA technologies, which are combinations 
of preventive and mitigating (PM) measures, to the attack scenarios in order to determine technology risk reduction 
potential based on terrorist attacks. Step 5 is 
summarized in Figure 37. The technologies under 
development are evaluated for three levels of 
integration to determine technology risk reduction 
potential and prioritization. The three different 
categories of technology integration are: 

• Stand-Alone 

• Integrated 

• Enhanced Integration 

The first analysis sub-set looks at each of the 
technologies operating independently in a stand-alone 
capability. This base case risk reduction potential 
does not consider how the technologies would work 
in an integrated fashion. The second analysis category 
is for the integrated Concept of Operations (CONOPS) case where the natural synergies of the technologies are 
considered in a concept of operation. In this CONOPS case, the technologies are assessed based on the current 
AvSSP project plan level of integration and levels of effort. In developing the CONOPS and considering the 
scenarios and threats, technology gaps are exposed. This natural exposure of gaps resulting from the logical flow of 
the models is another benefit of the LED methodology. Extra capabilities needed from the technologies to fill these 
gaps are determined. In this study, a new notional product, deemed the Collaborative Security Integrator (CSI), is 
developed. The CSI is the third category of integration and allows for the necessary data sharing and communication 
links plus additional attributes that are required to achieve even greater risk reduction potential from the 
technologies. The CSI category is defined as the enhanced technology integration capability for expressing terrorist 
attack risk reduction potential. 

The final step of this analysis effort is to evaluate the technology attack scenario risk reduction potential for the 
three levels of integration described above. The analysis results provide the NASA AvSSP with research portfolio 
prioritization recommendations based on attack scenario risk reduction. The risk reduction potential is defined as the 
difference between the ATS baseline risk (as it exists today) and the new terrorist attack scenario risk based on 
technology insertion. The analysis results provide the capability to rank order the risk reduction potential of 
applicable NASA research technologies. At this point in the analysis process, the second and final objective for this 
overall systems analysis effort of developing a decision support tool for the purposes of prioritizing NASA research 
in aviation security’ is accomplished. The ideal technology risk reduction potential and NASA AvSSP technology 
portfolio prioritization methodology for the three levels of integration is shown below in Figure 38. This risk 
reduction potential is termed ideal since it is exclusively based upon terrorist attack risk reduction. The technology 
nomenclature has been removed from the x-axis due to security classification in Figure 38. 



Figure 37 - Mapping Technologies to Attack Scenarios. 
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Figure 38 - Ideal Technology Risk Reduction Potential for the Three Categories 
and NASA AvSSP Technology Portfolio Prioritization. 


IV. Conclusions 

The NASA LaRC Systems Analysis and Concepts Directorate (SACD) includes a wide variety of 
capabilities, tools, methods, and experience areas of uncertainty quantification, propagation, decomposition, and 
management, risk analysis, robust design, reliability methods, and logic -evolved decision analysis within SACD. 
This group of disciplines referred to herein as Decision Support (DS) methods and tools, have been, or are currently 
being widely used within a broad spectrum of SACD applications. The DS disciplines are distinct from, and span 
across, the other technical disciplines typically used within SA such as structures, fluid and thermal dynamics, cost 
estimation, and reliability analysis. The DS methods and tools fill three critical roles in the SA process: 1) they 
provide the evaluations of uncertainty, risk, decision-making metrics within the analysis or design process, and 2) 
they provide feedback to the system analysis and design processes which helps to shape, define, and constrain the 
feasible analysis or design space and, perhaps, even the analysis and design paths used, and 3) they provide feedback 
to decision makers or stakeholders managing the analysis or design process so they can make informed choices. 
Many applications of these techniques were illustrated in this paper. 
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